Skip to content

ParparVM collector: long-shift codegen fix, live-set heap goal, and an arm64 parallel-mark run - #5717

Merged
shai-almog merged 25 commits into
masterfrom
parparvm-parallel-mark-arm64
Sep 7, 2026
Merged

ParparVM collector: long-shift codegen fix, live-set heap goal, and an arm64 parallel-mark run#5717
shai-almog merged 25 commits into
masterfrom
parparvm-parallel-mark-arm64

Conversation

@shai-almog

Copy link
Copy Markdown
Collaborator

Three separable changes, each with its rationale in the code rather than here.

1. 1L << n was computed as a 32-bit shift

BC_LSHL_EXPR / BC_LSHR_EXPR masked the shift count to 6 bits but left the left
operand alone, and the translator emits a long constant as a bare C literal. So
LCONST_1 arrived as an int:

1L << 31  ->  -2147483648      1L << 32  ->  1      1L << 33  ->  2

Shifting a long variable was always correct, which is how it survived.
BC_LUSHR_EXPR already cast, so this was fixed once for the unsigned shift and
not carried to its siblings. LongShift checks all three forms against a
reference built by repeated doubling.

2. Heap goal sized against the live set, not a constant

CN1_BIBOP_GC_TRIGGER_BYTES was the floor outright, so a process holding almost
nothing live still accumulated 24MB of garbage before collecting. Resident memory
tracked the trigger and nothing else (backend, /plaintext, 64 connections):

trigger 4MB 8MB 16MB 24MB 48MB
loaded RSS 30MB 49MB 68MB 98MB 102MB

Throughput and p99 across that sweep were flat inside noise, so the floor bought
footprint and no speed. The floor is now live-set + growth%, clamped to a 4MB
minimum, at all three clamp sites. An app with a real live set gets a larger
floor than the old constant. Measured after: 98MB -> 38MB loaded RSS, with
throughput and p50 improving.

Also adds cn1GcMutatorAssist (a thread parked on the run-ahead cap marks a
batch instead of sleeping). It registers in gcMarkActiveWorkers before
releasing the worklist mutex so mark termination cannot fire while it holds a
batch, and assists only the parallel path. It is inert until the mark pool is
enabled.

3. Re-test parallel marking on arm64

gcMarkResolveThreadCount forces one marker behind #elif 1. The comment reads
as a standing verdict that arm64 corrupts the heap with the pool on. The history
is narrower - inside #5327:

Jul 3   default parallel marking to serial ("isolation experiment")  <- the comment
Jul 4   gcMarkObject must reject freed BiBOP slots
Jul 5   SATB write barrier, closing the concurrent-mark cross-thread race
Jul 5   grace-subtree drain before sweep; belt pass; looped final mark
Jul 6   object-bearing frameless OFF - "unsound under conservative GC on arm64"
Jul 10  clazz-registry invariant (arm64 SIGSEGV)

The conclusion was drawn before every mechanism that makes concurrent marking
sound, the SATB barrier included. A different arm64 heap corruptor was found on
Jul 6. Parallel marking was never re-tested afterwards, and
gcMarkDrainParallel / gcMarkObject / gcMarkFlushLocal /
gcMarkWorklistPush have all been reworked since.

Cost of leaving it: on GcPause (one mutator, 20M short-lived objects, 4096-node
live set) the worst pause is 2.2-3.3s with one marker, 0.3-0.9s with four,
against Go's 20ms on the identical loop. Median and p99 already match Go.

Not reproduced locally: GcPause ran clean with four markers on an aarch64
guest, and a multi-threaded stress ran clean too - but that stress also passes
with the SATB barrier compiled out, so it is not a valid detector and is
deliberately not included here.

No defaults change. The pool is enabled for the new workflow only, via the
CN1_TEST_EXTRA_CFLAGS hook. Matrix: 1 marker arm64 (control), 4 markers arm64
(the question), 4 markers x64 (attribution).

Verification

GC suite (GcHeapIntegrity, GcOverflowSpiral, GcUncooperativeThread,
LargeArrayGc, LowMemoryThrottle, BibopPageFloor) 6/6 locally on this
branch. Copyright and control-character gates pass.

🤖 Generated with Claude Code

shai-almog and others added 3 commits September 6, 2026 10:00
BC_LSHL_EXPR and BC_LSHR_EXPR masked the shift count to 6 bits but left the
left operand alone, and the translator emits a long constant as a bare C
literal. LCONST_1 therefore arrived as an int and the whole shift was 32 bits
wide, so shifting a long constant by 31 or more silently produced the wrong
number:

    1L << 31  gave -2147483648   (int overflow, then sign-extended)
    1L << 32  gave 1             (int shift counts are masked to 5 bits)
    1L << 33  gave 2

Shifting a long VARIABLE was always correct, because the variable carries
JAVA_LONG into the macro where the constant does not -- which is how this
survived. BC_LUSHR_EXPR already cast, so the same bug was fixed once for the
unsigned shift and never carried across to its two siblings.

Found from a benchmark histogram whose bucket labels came out negative, which
is impossible for the long expression that computes them. LongShift checks all
three forms against a reference built by repeated doubling, so the check cannot
share a shift bug with what it is checking.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
CN1_BIBOP_GC_TRIGGER_BYTES was the floor outright, so a process holding almost
nothing live still let 24MB of garbage accumulate before collecting and the
BiBOP page pool sized itself to that. Resident memory tracked the trigger almost
linearly and nothing else -- measured on the backend, /plaintext at 64
connections:

    trigger  4MB -> 30MB RSS      trigger 24MB -> 98MB RSS
    trigger  8MB -> 49MB RSS      trigger 48MB -> 102MB RSS
    trigger 16MB -> 68MB RSS

Throughput and p99 across that sweep were flat inside run-to-run noise, so the
24MB floor was buying footprint and no speed. Every modern collector sizes the
next heap against the LIVE set instead -- Go's GOGC=100 means "collect when the
heap reaches twice what survived" -- which is why a Go server holding nothing
live sits at 6-17MB where this sat at 98MB.

The floor is now the live set plus CN1_BIBOP_HEAP_GROWTH_PERCENT of it, never
below CN1_BIBOP_GC_MIN_TRIGGER_BYTES, applied at all three clamp sites (shrink,
growth ceiling, low memory). It is not merely smaller: an application with a
real live set gets a LARGER floor than the old constant (20MB live at 100%
growth asks for 40MB where the constant gave 24MB), so this is more generous
exactly where the old rule was stingy and tighter only where it was wasteful.

Measured after: 98MB -> 38MB loaded RSS on the same route, with throughput and
p50 improving rather than regressing.

Also adds cn1GcMutatorAssist: a thread parked on the run-ahead cap marks a batch
instead of sleeping, which is Go's mutator assist in the one place this had a
sleep-until-the-cycle-ends park. It registers in gcMarkActiveWorkers before
releasing the worklist mutex so the "last worker out declares done" termination
cannot fire while it holds a batch, refuses to recurse when the thread is
already inside a mark, and assists only the parallel path -- the serial drain
touches the worklist without the mutex. It is therefore INERT until the mark
pool is enabled; see the CI workflow added separately.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
gcMarkResolveThreadCount compiles the mark pool out unconditionally. The comment
there is not a tuning note: it records that arm64 Linux still corrupted the heap
with the pool enabled after one acquire-load fix, and that a second ordering
hole was never located. The pool has been dead code since, so the collector
marks on one thread however many cores it has.

What that costs is measurable. On the GcPause benchmark added here -- one
mutator, 20M short-lived objects, a 4096-node live set, every iteration timed
into a log2 histogram -- the worst mutator pause is 2.2-3.3s with one marker and
0.3-0.9s with four, against Go's 20ms on the identical loop. Median and p99 are
identical to Go's at 32ns/64ns, so this is not a throughput deficit, it is a
stop that lasts seconds.

Reproducing the corruption is what this workflow is for, because it could not be
reproduced on an Apple-silicon podman guest. GcPause ran clean with four markers.
A multi-threaded stress with graph rewiring and cross-thread resurrection ran
clean too -- but that result is worthless, because the same stress also passes
with the SATB write barrier compiled out (-DCN1_DISABLE_SATB), which it must
not, and passes under -DCN1_GC_VERIFY at 6.6M references checked. A four-CPU
hypervisor guest is a weak generator of the store interleavings a
memory-ordering bug needs. These runners are native arm64 hardware, which is
where the corruption was actually seen.

The matrix runs one marker as the control, four markers on arm64 as the
question, and four markers on x64 so a failure can be attributed to the
architecture rather than to the pool itself.

No default changes. The marker count is raised only for this workflow, through
the CN1_TEST_EXTRA_CFLAGS hook the GC tests now honour; unset, every test
compiles exactly as before.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 6, 2026

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review Completed 2026-09-07T05:43:27.507236Z 5ad2848 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3ce45617b4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread vm/ByteCodeTranslator/src/cn1_globals.m
Comment thread .github/workflows/parparvm-parallel-mark.yml
Comment thread vm/ByteCodeTranslator/src/cn1_globals.m Outdated
@shai-almog

shai-almog commented Sep 6, 2026

Copy link
Copy Markdown
Collaborator Author

Compared 166 screenshots: 166 matched.
Native Windows port, REAL shipping pipeline: the hellocodenameone screenshot suite rendered by a binary CROSS-COMPILED on Linux (clang-cl + xwin, WebView2 linked) and RUN on a Windows x64 runner. Compared against the in-repo baseline in scripts/windows/screenshots.

Benchmark Results

Detailed Performance Metrics

Metric Duration
SIMD kernel backend SSE2 (x64) / NEON (arm64) native kernels
SIMD int-add (64K x300) java 63ms / native 4ms = 15.7x speedup
SIMD float-mul (64K x300) java 63ms / native 5ms = 12.6x speedup
SIMD kernel correctness PASS (native result == scalar reference)
Base64 native bridge unavailable (CN1 + SIMD + image benchmarks only)
Base64 payload size 8192 bytes
Base64 benchmark iterations 6000
Base64 SIMD byte path gated to scalar (CPU autovectorizes scalar; explicit SIMD not beneficial here)
Base64 CN1 encode 219.000 ms
Base64 CN1 decode 132.000 ms
Base64 SIMD encode 103.000 ms
Base64 encode ratio (SIMD/CN1) 0.470x (53.0% faster)
Base64 SIMD decode 100.000 ms
Base64 decode ratio (SIMD/CN1) 0.758x (24.2% faster)
Image encode benchmark iterations 100
Image createMask (SIMD off) 29.000 ms
Image createMask (SIMD on) 5.000 ms
Image createMask ratio (SIMD on/off) 0.172x (82.8% faster)
Image applyMask (SIMD off) 45.000 ms
Image applyMask (SIMD on) 76.000 ms
Image applyMask ratio (SIMD on/off) 1.689x (68.9% slower)
Image modifyAlpha (SIMD off) 50.000 ms
Image modifyAlpha (SIMD on) 52.000 ms
Image modifyAlpha ratio (SIMD on/off) 1.040x (4.0% slower)
Image modifyAlpha removeColor (SIMD off) 51.000 ms
Image modifyAlpha removeColor (SIMD on) 61.000 ms
Image modifyAlpha removeColor ratio (SIMD on/off) 1.196x (19.6% slower)

@shai-almog

shai-almog commented Sep 6, 2026

Copy link
Copy Markdown
Collaborator Author

Compared 166 screenshots: 166 matched.
Native Windows port (x64 / Intel-AMD): full hellocodenameone screenshot suite rendered offscreen with Direct2D/DirectWrite, plus the real benchmarks (base64 native/CN1/SIMD, image createMask/applyMask/modifyAlpha/PNG/JPEG, SSE2 SIMD kernels). Compared against the in-repo baseline in scripts/windows/screenshots.

Benchmark Results

Detailed Performance Metrics

Metric Duration
SIMD kernel backend SSE2 (x64) / NEON (arm64) native kernels
SIMD int-add (64K x300) java 63ms / native 4ms = 15.7x speedup
SIMD float-mul (64K x300) java 106ms / native 4ms = 26.5x speedup
SIMD kernel correctness PASS (native result == scalar reference)
Base64 native bridge unavailable (CN1 + SIMD + image benchmarks only)
Base64 payload size 8192 bytes
Base64 benchmark iterations 6000
Base64 SIMD byte path gated to scalar (CPU autovectorizes scalar; explicit SIMD not beneficial here)
Base64 CN1 encode 214.000 ms
Base64 CN1 decode 141.000 ms
Base64 SIMD encode 101.000 ms
Base64 encode ratio (SIMD/CN1) 0.472x (52.8% faster)
Base64 SIMD decode 98.000 ms
Base64 decode ratio (SIMD/CN1) 0.695x (30.5% faster)
Image encode benchmark iterations 100
Image createMask (SIMD off) 9.000 ms
Image createMask (SIMD on) 23.000 ms
Image createMask ratio (SIMD on/off) 2.556x (155.6% slower)
Image applyMask (SIMD off) 47.000 ms
Image applyMask (SIMD on) 68.000 ms
Image applyMask ratio (SIMD on/off) 1.447x (44.7% slower)
Image modifyAlpha (SIMD off) 53.000 ms
Image modifyAlpha (SIMD on) 92.000 ms
Image modifyAlpha ratio (SIMD on/off) 1.736x (73.6% slower)
Image modifyAlpha removeColor (SIMD off) 75.000 ms
Image modifyAlpha removeColor (SIMD on) 38.000 ms
Image modifyAlpha removeColor ratio (SIMD on/off) 0.507x (49.3% faster)

@github-actions

github-actions Bot commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

Cloudflare Preview

@github-actions

github-actions Bot commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

✅ Continuous Quality Report

Test & Coverage

Static Analysis

  • SpotBugs [Report archive]
    • ByteCodeTranslator: 0 findings (no issues)
    • android: 0 findings (no issues)
    • build-hint-catalog: 0 findings (no issues)
    • build-hint-tools: 0 findings (no issues)
    • codenameone-maven-plugin: 0 findings (no issues)
    • core-unittests: 0 findings (no issues)
    • ios: 0 findings (no issues)
  • PMD: 0 findings (no issues) [Report archive]
  • Checkstyle: 0 findings (no issues) [Report archive]

Generated automatically by the PR CI workflow.

@shai-almog

shai-almog commented Sep 6, 2026

Copy link
Copy Markdown
Collaborator Author

Compared 166 screenshots: 166 matched.
Native Linux port (x64), GTK3/Cairo/Pango, ParparVM bytecode-to-C (no JVM): the hellocodenameone screenshot suite rendered by a native ELF built + run on the GitHub x64 runner. Baseline: scripts/linux/screenshots.

@shai-almog

shai-almog commented Sep 6, 2026

Copy link
Copy Markdown
Collaborator Author

Compared 166 screenshots: 166 matched.
Native Linux port (arm64), GTK3/Cairo/Pango, ParparVM bytecode-to-C (no JVM): the hellocodenameone screenshot suite rendered by a native ELF built + run on the GitHub arm64 runner. Baseline: scripts/linux/screenshots-arm.

@shai-almog

shai-almog commented Sep 6, 2026

Copy link
Copy Markdown
Collaborator Author

Compared 166 screenshots: 166 matched.
Native Windows port (arm64 / Apple Silicon - Arm): full hellocodenameone screenshot suite rendered offscreen with Direct2D/DirectWrite, plus the real benchmarks (base64 native/CN1/SIMD, image createMask/applyMask/modifyAlpha/PNG/JPEG, NEON SIMD kernels). Compared against the in-repo baseline in scripts/windows/screenshots.

Benchmark Results

Detailed Performance Metrics

Metric Duration
SIMD kernel backend SSE2 (x64) / NEON (arm64) native kernels
SIMD int-add (64K x300) java 56ms / native 4ms = 14.0x speedup
SIMD float-mul (64K x300) java 57ms / native 3ms = 19.0x speedup
SIMD kernel correctness PASS (native result == scalar reference)
Base64 native bridge unavailable (CN1 + SIMD + image benchmarks only)
Base64 payload size 8192 bytes
Base64 benchmark iterations 6000
Base64 SIMD byte path gated to scalar (CPU autovectorizes scalar; explicit SIMD not beneficial here)
Base64 CN1 encode 268.000 ms
Base64 CN1 decode 157.000 ms
Base64 SIMD encode 65.000 ms
Base64 encode ratio (SIMD/CN1) 0.243x (75.7% faster)
Base64 SIMD decode 63.000 ms
Base64 decode ratio (SIMD/CN1) 0.401x (59.9% faster)
Image encode benchmark iterations 100
Image createMask (SIMD off) 7.000 ms
Image createMask (SIMD on) 3.000 ms
Image createMask ratio (SIMD on/off) 0.429x (57.1% faster)
Image applyMask (SIMD off) 25.000 ms
Image applyMask (SIMD on) 19.000 ms
Image applyMask ratio (SIMD on/off) 0.760x (24.0% faster)
Image modifyAlpha (SIMD off) 17.000 ms
Image modifyAlpha (SIMD on) 13.000 ms
Image modifyAlpha ratio (SIMD on/off) 0.765x (23.5% faster)
Image modifyAlpha removeColor (SIMD off) 21.000 ms
Image modifyAlpha removeColor (SIMD on) 13.000 ms
Image modifyAlpha removeColor ratio (SIMD on/off) 0.619x (38.1% faster)

…he floor

Four review findings, all of them correct.

1. The mutator went ACTIVE again in the middle of the GC handshake. threadActive
   is what tells the collector it may scan this thread's roots without stopping
   it, so raising it after the sleep -- when threadBlockedByGC may have gone up
   during that sleep and a conservative scan already be walking this stack --
   let the mutator, and the assist running mark functions on it, move the stack
   underneath the scan. It now waits the block out before reactivating, matching
   what the tail of the function already did.

2. The injected flags concatenated with the test's own: extraCFlags() had a
   leading space and no trailing one, so a fault-injection build asked for
   -DCN1_GC_MARK_THREADS=4-DCN1_GC_NO_FORCE_STOP -- one undeclared macro rather
   than two, which clang accepts. It is padded on both sides now, and the tests
   that pass their own flags go through cFlagsArg/objcFlagsArg.

3. The hook reached two of the six tests in the matrix, so four of them compiled
   the default single-marker collector and a green matrix would not have
   validated what the workflow claims. Worse than the review said: moving the
   hook into cmakeToolchainArgs fixed only two more, because the rest build
   their cmake command inline with hardcoded compilers. Found by probing with an
   invalid flag -- the build should have failed and did not. All six are wired
   now and the probe fails as it should, with the flag visible on the clang
   command line.

4. liveBytes is not the whole live set. The sweep walks the retired-page list
   and a page the major sweep splices out of a partial pool is deliberately
   withheld from policy statistics, so sizing the floor from one sample let it
   collapse toward the minimum while a large live heap sat on pages the cycle
   never looked at -- the collector would then retrace that heap every few
   megabytes. There is no registry-wide live count to use instead, so the floor
   now tracks a decaying high-water mark: a recent cycle that did see a large
   live set holds it up, and a genuinely small one walks it down in a few
   cycles.

Also fixes the workflow itself, which failed on all three arms including the
control: mvn package does not install, so -pl tests could not resolve the
translator. It now packages with -am and tests with -am, mirroring
parparvm-tests.yml.

GC suite 6/6 locally with the hook unset.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6791382eae

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread vm/ByteCodeTranslator/src/cn1_globals.m Outdated
@shai-almog

shai-almog commented Sep 6, 2026

Copy link
Copy Markdown
Collaborator Author

Compared 181 screenshots: 181 matched.
✅ JavaScript-port screenshot tests passed.

…ell as lower

Two CI failures and one review finding, all in the trigger floor.

GcSteadyState failed in CI with "Pinning the free-memory reading to 16MB did not
make the per-thread pending table fill, so this scenario and its twin measure
nothing" (pendingFullParks=0). That is the test asserting its own non-vacuity,
and it was right: the low-memory branch had been switched to the live-set floor,
which collects early enough under a 16MB pin that the pending table never fills.
Low memory mode is about surviving pressure rather than about footprint, so it
goes back to pinning the constant and the floor no longer applies there.

The floor also only ever lowered the trigger. A trigger that had adapted down
before the live set grew stayed down: 4MB against 20MB live keeps retracing that
live heap every 4MB of allocation, because the shrink branch compared in one
direction only. It now clamps upward to the floor as well, which is the only
path that corrects a trigger which adapted below what the live set justifies.

CompilerHelper carried no copyright header; the gate checks files a PR touches,
so modifying it required adding one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 169f2a1464

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread vm/ByteCodeTranslator/src/cn1_globals.m Outdated
Comment thread vm/ByteCodeTranslator/src/cn1_globals.m Outdated
@shai-almog

shai-almog commented Sep 6, 2026

Copy link
Copy Markdown
Collaborator Author

Compared 160 screenshots: 160 matched.
✅ Native Mac screenshot tests passed.

Benchmark Results

  • VM Translation Time: 0 seconds
  • Compilation Time: 276 seconds

Detailed Performance Metrics

Metric Duration
SIMD kernel backend SSE2 (x64) / NEON (arm64) native kernels
SIMD int-add (64K x300) java 82ms / native 4ms = 20.5x speedup
SIMD float-mul (64K x300) java 90ms / native 5ms = 18.0x speedup
SIMD kernel correctness PASS (native result == scalar reference)
Base64 native bridge unavailable (CN1 + SIMD + image benchmarks only)
Base64 payload size 8192 bytes
Base64 benchmark iterations 6000
Base64 SIMD byte path active (NEON-accelerated)
Base64 CN1 encode 166.000 ms
Base64 CN1 decode 96.000 ms
Image encode benchmark iterations 100
Image createMask (SIMD off) 6.000 ms
Image createMask (SIMD on) 2.000 ms
Image createMask ratio (SIMD on/off) 0.333x (66.7% faster)
Image applyMask (SIMD off) 45.000 ms
Image applyMask (SIMD on) 40.000 ms
Image applyMask ratio (SIMD on/off) 0.889x (11.1% faster)
Image modifyAlpha (SIMD off) 43.000 ms
Image modifyAlpha (SIMD on) 34.000 ms
Image modifyAlpha ratio (SIMD on/off) 0.791x (20.9% faster)
Image modifyAlpha removeColor (SIMD off) 42.000 ms
Image modifyAlpha removeColor (SIMD on) 32.000 ms
Image modifyAlpha removeColor ratio (SIMD on/off) 0.762x (23.8% faster)

@shai-almog

shai-almog commented Sep 6, 2026

Copy link
Copy Markdown
Collaborator Author

Compared 144 screenshots: 144 matched.
✅ Native Apple TV (tvOS, Metal) screenshot tests passed.

@shai-almog

shai-almog commented Sep 6, 2026

Copy link
Copy Markdown
Collaborator Author

Compared 148 screenshots: 148 matched.
✅ Native Mac screenshot tests passed.

Benchmark Results

  • VM Translation Time: 0 seconds
  • Compilation Time: 405 seconds

Detailed Performance Metrics

Metric Duration
SIMD kernel backend SSE2 (x64) / NEON (arm64) native kernels
SIMD int-add (64K x300) java 67ms / native 3ms = 22.3x speedup
SIMD float-mul (64K x300) java 53ms / native 3ms = 17.6x speedup
SIMD kernel correctness PASS (native result == scalar reference)
Base64 payload size 8192 bytes
Base64 benchmark iterations 6000
Base64 SIMD byte path active (NEON-accelerated)
Base64 CN1 encode 215.000 ms
Base64 CN1 decode 106.000 ms
Base64 native encode 1081.000 ms
Base64 encode ratio (CN1/native) 0.199x (80.1% faster)
Base64 native decode 558.000 ms
Base64 decode ratio (CN1/native) 0.190x (81.0% faster)
Base64 SIMD encode 57.000 ms
Base64 encode ratio (SIMD/CN1) 0.265x (73.5% faster)
Base64 SIMD decode 71.000 ms
Base64 decode ratio (SIMD/CN1) 0.670x (33.0% faster)
Base64 encode ratio (SIMD/native) 0.053x (94.7% faster)
Base64 decode ratio (SIMD/native) 0.127x (87.3% faster)
Image encode benchmark iterations 100
Image createMask (SIMD off) 17.000 ms
Image createMask (SIMD on) 6.000 ms
Image createMask ratio (SIMD on/off) 0.353x (64.7% faster)
Image applyMask (SIMD off) 65.000 ms
Image applyMask (SIMD on) 65.000 ms
Image applyMask ratio (SIMD on/off) 1.000x (0.0% slower)
Image modifyAlpha (SIMD off) 78.000 ms
Image modifyAlpha (SIMD on) 49.000 ms
Image modifyAlpha ratio (SIMD on/off) 0.628x (37.2% faster)
Image modifyAlpha removeColor (SIMD off) 58.000 ms
Image modifyAlpha removeColor (SIMD on) 52.000 ms
Image modifyAlpha removeColor ratio (SIMD on/off) 0.897x (10.3% faster)

@shai-almog

shai-almog commented Sep 6, 2026

Copy link
Copy Markdown
Collaborator Author

Compared 217 screenshots: 217 matched.
✅ Native Apple Watch (watchOS, Core Graphics) screenshot tests passed.

GcSteadyState failed itself in CI again on the previous commit: "Pinning the
free-memory reading to 16MB did not make the per-thread pending table fill, so
this scenario and its twin measure nothing", with cyclesOnDemand=5598 and 22331
volume parks. Reverting the low-memory branch to the constant was not enough --
the trigger was sitting at the 4MB minimum in the NORMAL path, so the collector
ran continuously and the pending table never filled.

Review found the same edge from the other side: liveBytes is a per-sweep sample
that systematically understates, because the sweep walks the retired list and
pages the major sweep splices out of a partial pool are withheld from policy
statistics. A decaying high-water delays that rather than fixing it -- about
twenty low-survival cycles and a stable 20MB live set is estimated at the
minimum, after which the collector retraces that heap every few megabytes. The
same review noted ProcessBudgetPacingIntegrationTest derives its 72MB static cap
floor from this constant, so lowering it can fail that test too.

CN1_BIBOP_GC_MIN_TRIGGER_BYTES therefore defaults to the old constant. The
proportional rule now only ever RAISES the floor: an application with a real
live set gets more headroom than the constant gave it, and nothing gets less. A
deployment that knows its live set is tiny defines it lower at build time -- the
backend does, and that is where the 98MB to 38MB resident-memory measurement
came from.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7269ca8b24

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread vm/ByteCodeTranslator/src/cn1_globals.m
Comment thread vm/ByteCodeTranslator/src/cn1_globals.m
Comment thread vm/ByteCodeTranslator/src/cn1_globals.m Outdated
@shai-almog

shai-almog commented Sep 6, 2026

Copy link
Copy Markdown
Collaborator Author

Compared 149 screenshots: 149 matched.
✅ Native iOS Metal screenshot tests passed.

Benchmark Results

  • VM Translation Time: 0 seconds
  • Compilation Time: 1582 seconds

Build and Run Timing

Metric Duration
Simulator Boot 102000 ms
Simulator Boot (Run) 1000 ms
App Install 19000 ms
App Launch 5000 ms
Test Execution 495000 ms

Detailed Performance Metrics

Metric Duration
SIMD kernel backend SSE2 (x64) / NEON (arm64) native kernels
SIMD int-add (64K x300) java 70ms / native 3ms = 23.3x speedup
SIMD float-mul (64K x300) java 73ms / native 3ms = 24.3x speedup
SIMD kernel correctness PASS (native result == scalar reference)
Base64 payload size 8192 bytes
Base64 benchmark iterations 6000
Base64 SIMD byte path active (NEON-accelerated)
Base64 CN1 encode 202.000 ms
Base64 CN1 decode 155.000 ms
Base64 native encode 496.000 ms
Base64 encode ratio (CN1/native) 0.407x (59.3% faster)
Base64 native decode 396.000 ms
Base64 decode ratio (CN1/native) 0.391x (60.9% faster)
Base64 SIMD encode 68.000 ms
Base64 encode ratio (SIMD/CN1) 0.337x (66.3% faster)
Base64 SIMD decode 65.000 ms
Base64 decode ratio (SIMD/CN1) 0.419x (58.1% faster)
Base64 encode ratio (SIMD/native) 0.137x (86.3% faster)
Base64 decode ratio (SIMD/native) 0.164x (83.6% faster)
Image encode benchmark iterations 100
Image createMask (SIMD off) 7.000 ms
Image createMask (SIMD on) 2.000 ms
Image createMask ratio (SIMD on/off) 0.286x (71.4% faster)
Image applyMask (SIMD off) 39.000 ms
Image applyMask (SIMD on) 26.000 ms
Image applyMask ratio (SIMD on/off) 0.667x (33.3% faster)
Image modifyAlpha (SIMD off) 30.000 ms
Image modifyAlpha (SIMD on) 26.000 ms
Image modifyAlpha ratio (SIMD on/off) 0.867x (13.3% faster)
Image modifyAlpha removeColor (SIMD off) 35.000 ms
Image modifyAlpha removeColor (SIMD on) 26.000 ms
Image modifyAlpha removeColor ratio (SIMD on/off) 0.743x (25.7% faster)

@github-actions

github-actions Bot commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

✅ ByteCodeTranslator Quality Report

Test & Coverage

  • Tests: 562 total, 0 failed, 54 skipped

Benchmark Results

  • Execution Time: 20878 ms

  • Hotspots (Top 20 sampled methods):

    • 23.43% com.codename1.tools.translator.Parser.addToConstantPool (429 samples)
    • 6.94% java.util.ArrayList.indexOf (127 samples)
    • 3.99% com.codename1.tools.translator.ByteCodeClass.hasDeclaredMethod (73 samples)
    • 3.93% com.codename1.tools.translator.ByteCodeClass.fillVirtualMethodTable (72 samples)
    • 2.89% com.codename1.tools.translator.BytecodeMethod.optimize (53 samples)
    • 2.68% com.codename1.tools.translator.Parser.cn1EnsureSubclassIndex (49 samples)
    • 2.57% java.lang.StringBuilder.append (47 samples)
    • 2.40% com.codename1.tools.translator.BytecodeMethod.equals (44 samples)
    • 2.29% java.lang.Object.hashCode (42 samples)
    • 1.86% com.codename1.tools.translator.Parser.classIndex (34 samples)
    • 1.86% org.objectweb.asm.tree.analysis.Analyzer.findSubroutine (34 samples)
    • 1.69% org.objectweb.asm.tree.analysis.Analyzer.analyze (31 samples)
    • 1.37% com.codename1.tools.translator.bytecodes.Invoke.resolveDirectTarget (25 samples)
    • 1.31% org.objectweb.asm.ClassReader.readCode (24 samples)
    • 1.20% com.codename1.tools.translator.BytecodeMethod.appendCMethodPrefix (22 samples)
    • 1.09% java.lang.System.identityHashCode (20 samples)
    • 1.09% java.lang.String.equals (20 samples)
    • 0.98% com.codename1.tools.translator.bytecodes.Invoke.findMethodUp (18 samples)
    • 0.87% java.util.HashMap.hash (16 samples)
    • 0.82% com.codename1.tools.translator.Parser.resolveDupForms (15 samples)
  • ⚠️ Coverage report not generated.

Static Analysis

  • ✅ SpotBugs: no findings (report was not generated by the build).
  • ⚠️ PMD report not generated.
  • ⚠️ Checkstyle report not generated.

Generated automatically by the PR CI workflow.

shai-almog and others added 2 commits September 6, 2026 20:03
…ctly-once

Three review findings on the GC/profiling changes, all of them real.

cn1GcMutatorAssist declared marking complete on the worker count alone. A
regular marker only decrements after re-checking the worklist at the top of
its loop, so it can never finish with work outstanding; the assist decrements
immediately after gcMarkFlushLocal, which may have just published children
this batch discovered. With CN1_GC_MARK_THREADS > 1 the assist could take the
count to zero against a non-empty worklist, end the parallel drain with
reachable subtrees unscanned, and let the following sweep reclaim them.
Termination now requires an empty worklist as well.

The allocation profile hooked only codenameOneGcMalloc and
cn1BibopFastAllocNoZero. CN1_FAST_NEW calls cn1BibopFastAlloc -- a different
function -- and cn1AllocFused and cn1FusedLatin1Begin are two further entry
points, so a build routing through any of them under-reported silently. All
four entry points now record, and each hook sits on the success return rather
than at entry: a fast path returning 0 falls back to codenameOneGcMalloc, so
the old entry-side hook counted that allocation twice. cn1BibopAlloc stays
unhooked on purpose, being an internal callee of three of them. The fused
latin1 path attributes the String and the byte[] payload separately, since
the profile is read to find out what is being allocated.

The per-class average divided by count|1, which for an even count changed the
divisor rather than guarding zero: two allocations reported bytes/3.

Verified by compiling cn1_globals.m in both the default and CN1_GC_CONFORM
configurations, each against a pre-change baseline, with undeclared-identifier
probes at the new header and .m hook sites to prove the regions are really
compiled rather than preprocessed away.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The profile counts requested bytes at allocation time; allocatedKb accumulates
BiBOP slot bytes at GC cycle boundaries only. A previous version of the profile
double-counted fast-path fallbacks and still agreed with allocatedKb to within
2%, because the over- and under-counting cancelled. The note exists so that
agreement is not read as verification a second time.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1493d4b0fa

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread vm/ByteCodeTranslator/src/cn1_globals.m Outdated
Comment thread vm/ByteCodeTranslator/src/cn1_globals.m
…file omitting classes silently

Two review findings, both real.

The live-set trigger floor was only applied inside the two survival branches,
which cover >= 25% and <= 20% and nothing between. A trigger that adapted down
before the live set grew therefore sat uncorrected for as long as survival
stayed at 21-24%, retracing a live heap it no longer had the headroom for. The
floor is a property of the live set rather than of the survival rate, so it is
now enforced after the branches. Only the raising direction is taken, which is
what the <= 20% branch already did, so the dead band gets the behaviour the
branches either side of it already had.

The allocation profile dropped any class whose id fell outside its table, and
dropped it from the total as well as the ranking -- so a profile missing its
hottest class read exactly like one that found nothing there. Class ids number
scalar classes and array classes in one contiguous range, so an app well inside
the limit on scalar classes can still put its array classes past it. The bound
is raised past any plausible translated app, and anything still outside is
counted, added to the total, and printed as an explicit row naming the highest
id seen. This file already had one instrument that lied by omission; it should
not have a second.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 70dad8d317

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread vm/ByteCodeTranslator/src/cn1_globals.m Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 05b6bec486

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread .github/workflows/parparvm-parallel-mark.yml Outdated
shai-almog and others added 2 commits September 6, 2026 21:03
The paths filter matched Gc*.java, which covers three of the six tests the
matrix runs: LargeArrayGcIntegrationTest, LowMemoryThrottleIntegrationTest and
BibopPageFloorIntegrationTest did not match, and neither did CompilerHelper,
which supplies the CN1_TEST_EXTRA_CFLAGS every arm depends on to select its
marker count. A change to any of them could land without the workflow that
exercises them ever running -- and a CompilerHelper regression is not
hypothetical: a central fix in this branch reached three of six tests and the
gap was found by an invalid-flag probe rather than by CI.

Replaced with one entry per test plus the helper, and a note to keep the list
and the -Dtest list in sync. The glob is what failed; an explicit list at least
fails visibly when the two drift.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
BibopPageFloorIntegrationTest failed on the 1-marker arm -- the default
configuration -- and it is this branch's regression rather than a flake.

Master has no live-set floor: at survival <= 20% it halves the trigger toward
the 24MB constant and stops. This branch raises the trigger to a floor derived
from bibopLiveHighWater instead, and cn1BibopTrimFreePool, which hands surplus
pages back to the OS, runs ONLY at the end of a sweep. Those two facts latch:
the test drops a 196MB live set, the floor still describes the live set that has
just gone, the raised trigger puts the next sweep far away, and the decay that
would lower the high-water runs once per sweep -- so a high floor buys itself
the scarcity of cycles that keeps it high. Across runs of one commit the test
returned 6%, 73% and 91% of its pages; bimodal, which is a sweep happening or
not, rather than a budget being crept over.

The high-water exists to damp the floor on the way UP, where a per-sweep sample
understates. It was being applied in both directions, which is the actual
defect. Growth is still damped through it; a collapse is now believed
immediately, with one minimum-trigger of slack so a sample that merely dips does
not slam the floor to the minimum. Evaluated over the cases that matter: a
dropped live set falls from a latched 192MB to 48MB on the very next cycle,
while a steady live set -- the server's case, and the one the floor was
introduced for -- and a growing one are bit for bit what they were.

Three local repeats pass at 73% released, which is what the run before this
change also returned: the latch does not reproduce on this host, so those runs
show the common case is unchanged and nothing more. The arm64 matrix is the
environment that discriminates.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@shai-almog

shai-almog commented Sep 6, 2026

Copy link
Copy Markdown
Collaborator Author

Compared 143 screenshots: 143 matched.
✅ Native iOS screenshot tests passed.

Benchmark Results

  • VM Translation Time: 0 seconds
  • Compilation Time: 1323 seconds

Build and Run Timing

Metric Duration
Simulator Boot 64000 ms
Simulator Boot (Run) 0 ms
App Install 15000 ms
App Launch 4000 ms
Test Execution 410000 ms

Detailed Performance Metrics

Metric Duration
SIMD kernel backend SSE2 (x64) / NEON (arm64) native kernels
SIMD int-add (64K x300) java 56ms / native 3ms = 18.6x speedup
SIMD float-mul (64K x300) java 55ms / native 3ms = 18.3x speedup
SIMD kernel correctness PASS (native result == scalar reference)
Base64 payload size 8192 bytes
Base64 benchmark iterations 6000
Base64 SIMD byte path active (NEON-accelerated)
Base64 CN1 encode 267.000 ms
Base64 CN1 decode 96.000 ms
Base64 native encode 602.000 ms
Base64 encode ratio (CN1/native) 0.444x (55.6% faster)
Base64 native decode 542.000 ms
Base64 decode ratio (CN1/native) 0.177x (82.3% faster)
Base64 SIMD encode 50.000 ms
Base64 encode ratio (SIMD/CN1) 0.187x (81.3% faster)
Base64 SIMD decode 150.000 ms
Base64 decode ratio (SIMD/CN1) 1.563x (56.3% slower)
Base64 encode ratio (SIMD/native) 0.083x (91.7% faster)
Base64 decode ratio (SIMD/native) 0.277x (72.3% faster)
Image encode benchmark iterations 100
Image createMask (SIMD off) 7.000 ms
Image createMask (SIMD on) 2.000 ms
Image createMask ratio (SIMD on/off) 0.286x (71.4% faster)
Image applyMask (SIMD off) 77.000 ms
Image applyMask (SIMD on) 154.000 ms
Image applyMask ratio (SIMD on/off) 2.000x (100.0% slower)
Image modifyAlpha (SIMD off) 100.000 ms
Image modifyAlpha (SIMD on) 155.000 ms
Image modifyAlpha ratio (SIMD on/off) 1.550x (55.0% slower)
Image modifyAlpha removeColor (SIMD off) 343.000 ms
Image modifyAlpha removeColor (SIMD on) 28.000 ms
Image modifyAlpha removeColor ratio (SIMD on/off) 0.082x (91.8% faster)

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 409ee59b2d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread vm/ByteCodeTranslator/src/cn1_globals.m Outdated
Comment thread vm/ByteCodeTranslator/src/cn1_globals.m Outdated
The previous commit let a low liveBytes take the floor down immediately, and
that is too credulous. liveBytes is not a measurement of the live set, it is the
part of it this sweep sampled: pages the major sweep splices out of a partial
pool are withheld from the policy numbers on purpose. Both halves of the ratio
skip the same pages, so survival is unaffected, but the ABSOLUTE figure
understates by whatever was withheld -- and a stable 100MB heap whose objects
happen to sit on excluded pages reports a near-zero live set. Collapsing on that
would halve the trigger against a heap that never shrank and retrace it every
cycle, which is the cost the floor exists to avoid and worse than the latch it
replaced: the latch at least erred toward collecting less.

The sweep now carries how much it withheld, and a collapse is believed only when
that is at most an eighth of what it did measure -- if the pages it could not see
are a small part of the heap, the objects hiding on them cannot be many. The
threshold is deliberately strict in that direction because the two errors are not
symmetric: a wrong yes retraces a live heap every cycle, a wrong no just leaves
the decay to walk the high-water down as it did before.

Evaluated over all three cases. A real collapse still drops the floor from a
latched 192MB to 48MB on the next cycle. A stable 100MB heap sampled at 1MB
through an incomplete sweep now holds at 175, 153, 133MB and decays; with the
gate removed the same input collapses it to 50MB, which is the reported failure
reproduced.

Also publishes the allocation-size filter through pthread_once. The flag and the
pointer beside it were read and written by every mutator's first allocation, and
the visible form of that race is the wrong one for an instrument: a thread
seeing the flag set before the pointer is published reads a null name and drops
its early samples, losing exactly the allocations a start-up question asks about.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 78150e72d8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread vm/ByteCodeTranslator/src/cn1_globals.h Outdated
Comment thread vm/ByteCodeTranslator/src/cn1_globals.m Outdated
…ters atomically

Java defines << as two's-complement wraparound. C does not: a signed left shift
whose result is not representable is UNDEFINED, so 1L << 63, 1 << 31 and every
negative left operand were cases the optimizer was entitled to assume could not
happen. The earlier fix here added the JAVA_LONG cast that stopped the
translator's bare int literal making 1L << 32 evaluate to 1, and left the
undefined shift underneath it. All four LEFT shifts now go through the unsigned
type of the same width and convert back, which is Java's answer with no
undefined step.

The signed RIGHT shifts are deliberately untouched: a negative >> n is
implementation-defined rather than undefined, and every compiler this VM builds
with defines it as the arithmetic shift Java specifies.

Checked by evaluating the real macro text, lifted out of the header rather than
retyped, against Java's values under -O2 with -fsanitize=undefined and
-fno-sanitize-recover. The old macro is reported as "left shift of 1 by 63
places cannot be represented in type JAVA_LONG" and the new one is clean, so the
check is not vacuous. LongShift now runs to 63 and covers negative operands and
int shifts, since stopping at 34 exercised the literal bug and nothing about the
shift; its doubling reference agrees with the sanitizer harness on every case.

Separately, cn1AllocProfClass is atomic like the counters beside it. Several
mutators allocating one class wrote the slot at once -- same-value concurrent
writes are still a race -- and the atexit report reads it while allocation
continues, where a torn value would be dereferenced to print a class name.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9bb011e84e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread vm/ByteCodeTranslator/src/cn1_globals.m Outdated
Load-then-store is a read-modify-write: two threads reporting out-of-range
classes could both pass the comparison and let the smaller id land last. The
number exists to tell a reader how far the table must grow, so understating it
sends the next run back with a bound that is still too small -- the same failure
the row was added to prevent.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f28e2a827c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread vm/ByteCodeTranslator/src/cn1_globals.m Outdated
Comment thread vm/ByteCodeTranslator/src/cn1_globals.m Outdated
The coverage gate added in the previous commit was backwards in both directions,
and the review is right about why. It tested how many bytes the sweep withheld
from the survival ratio. An ORDINARY sweep withholds nothing -- so it was called
complete precisely when it saw least, because it walks retired pages alone and
every live object on a partial page is invisible to it. A MAJOR sweep withholds
exactly the partial pages it spliced in -- so it was called incomplete when it
had in fact just walked the entire heap. A small liveBytes from an ordinary
sweep is not a small live set, it is a small sample, and collapsing the floor on
it would retrace a stable heap every cycle.

Only a major sweep can say anything about the size of the live set, so that is
now the gate. And when it speaks it is asked for the WHOLE live set: liveBytes
plus what was withheld. That figure was already being computed page by page and
thrown away; the survival ratio still excludes those slots exactly as before,
because the reason for excluding them -- a spliced partial page is a deep sample
that drags the ratio down -- is about the RATIO and was never about the total.

Evaluated over the three cases. A stable 100MB live set with 99MB of it on
partial pages holds at 175, 153, 133MB across ordinary sweeps instead of
collapsing; the major sweep that follows reads the complete 100MB and puts the
floor at 192MB; and a real collapse seen by a major sweep still releases the
latch to 48MB.

Also resolves the assist's A/B flag through pthread_once. It ran on every paced
mutator, so with more than one marker several reach their first assist together
and a function-local static was read and written by all of them at once -- a
data race in exactly the parallel configuration the new workflow exists to
exercise. Compiles clean with CN1_GC_MARK_THREADS=4, and an undeclared-identifier
probe at the new call site confirms that region is really compiled there.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3d0209c5de

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread vm/ByteCodeTranslator/src/cn1_globals.m Outdated
Fourth finding on the same twenty lines, and the review is right again. The
figure the floor was derived from is not the live set and cannot be made into
one here. An ordinary sweep walks retired pages alone, so every live object on a
partial page is invisible to it. A major sweep splices the partial pools in, but
mutator-owned current pages are never swept at all -- an invariant of the sweep,
not an omission -- and the legacy heap above CN1_BIBOP_MAX_OBJECT is not in
BiBOP pages and never reaches these counters at all.

Three attempts to build a policy on it were each unsound in a different
direction: it latched a dropped live set at 192MB and starved the sweeps that
hand pages back, which is BibopPageFloorIntegrationTest returning 6% of its
pages against 91% on a luckier run of the same commit; then it believed a
partial sample as a collapse; then its coverage gate called an ordinary sweep
complete precisely when it saw least. Each fix was sound against the case that
prompted it and wrong about the next one, which is what building on an
unmeasurable quantity looks like from the inside.

So the floor goes, along with the decaying high-water, the dead-band clamp, the
coverage gate and the sweep plumbing that fed them. Nothing measured is lost:
the footprint result this work was for -- 98MB to 38MB on the backend -- is
attributed to the MINIMUM, which is a build-time constant and needs no estimate.
A deployment that knows its live set is small defines
CN1_BIBOP_GC_MIN_TRIGGER_BYTES lower and the shrink path walks the trigger down
to it.

The trigger policy is now master's, verified by diffing the function against
origin/master with comments stripped: the only difference is that the shrink
target and the ceiling's lower bound read CN1_BIBOP_GC_MIN_TRIGGER_BYTES instead
of the hard CN1_BIBOP_GC_TRIGGER_BYTES, and that constant defaults to the same
24MB, so a stock build behaves exactly as master does. This also removes the
21-24% dead band an earlier round reported: that finding was real, but it was a
real defect in a mechanism that should not have existed, and with no floor there
is no dead band to fall into.

Compiles clean in the default, CN1_GC_CONFORM, CN1_DISABLE_BIBOP and
CN1_GC_MARK_THREADS=4 configurations. BibopPageFloor, LowMemoryThrottle and
LargeArrayGc pass locally, the first returning 73% of its pages.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 99ba28bc94

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread vm/ByteCodeTranslator/src/cn1_globals.m
"Quiet" describes a cycle that ran without the application allocating much, and
what counts as much is relative to how much allocation it takes to start a cycle
at all. The cutoff was a fixed quarter of the 24MB default, which breaks as soon
as the trigger is lower than it: a deployment that sets
CN1_BIBOP_GC_MIN_TRIGGER_BYTES to 4MB -- the configuration this branch added and
the server uses -- collects every 4 to 6MB, every one of those ordinary
allocation-driven cycles falls under a 6MB cutoff, and every single collection
then splices every partial pool. That is the O(all pages) sweep issue 5425
removed, handed back to exactly the workload that reported it.

The cutoff is now a quarter of the live trigger, capped by the old constant so
nothing gets looser than it was. At the 24MB default it is 6MB and unchanged; at
a 4MB trigger it is 1MB, so an ordinary crossing is no longer mistaken for
silence; at a grown 192MB trigger the constant still caps it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b1d7f46014

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread vm/ByteCodeTranslator/src/cn1_globals.m
shai-almog and others added 2 commits September 6, 2026 22:11
The branch assigned CN1_BIBOP_GC_TRIGGER_BYTES unconditionally. On master that
could only ever bring the trigger down, because nothing there put it below 24MB.
This branch lets a deployment define CN1_BIBOP_GC_MIN_TRIGGER_BYTES lower -- the
server uses 4MB -- and then the same assignment RAISES a trigger the low-survival
path had already shrunk. An OS memory warning would postpone collection at the
moment headroom is scarcest, and lift the pacing cap derived from the trigger
along with it.

Pinning down to the constant is still the intent and is untouched at the default
minimum: GcSteadyState pins the free-memory reading to 16MB to make the
per-thread pending table fill and runs with the default, so its trigger is at or
above the constant and takes exactly the branch it always did. A 192MB trigger
still drops to 24MB; a 4MB one now stays 4MB instead of tripling.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…lains itself

BibopPageFloorIntegrationTest fails here intermittently -- twice in six runs, on
the 1-marker arm once and the 4-marker arm once, arm64 only, never x64 -- and
the failure is bimodal rather than marginal: 6-7% of a dropped live set's pages
handed back on a bad run against 91% on a good one. Bimodal is a major sweep
happening or not happening, because cn1BibopTrimFreePool runs only at the end of
one. So the question a failing run has to answer is whether any major sweep ran
after the live set was dropped and what it spliced, and [MAJOR-SWEEP] and
[PAGE-RELEASE] answer exactly that.

Guessing has been tried. The failure was attributed to this branch's live-set
floor on a plausible mechanism and two data points; the floor has since been
removed and the test failed again without it, so that diagnosis was wrong. A
second guess -- that Linux was getting MADV_FREE, which does not reduce RSS
promptly, where Darwin gets MADV_FREE_REUSABLE -- is also wrong: the release
path is guarded, and Linux takes the MADV_DONTNEED branch, which frees
immediately. The cause is genuinely unknown, which is the argument for
instrumenting rather than theorising a third time.

Safe to leave on. The tracer is gated behind getenv and only prints, so no
collection decision changes, and it writes to stderr, which runVm deliberately
routes to the console instead of merging into stdout -- a merged tracer line can
land mid-marker and silently drop a row from the table the assertions parse.
Verified locally: the test still passes at 73% with the markers intact, and the
run emits 53 [MAJOR-SWEEP] and 4 [PAGE-RELEASE] lines, so the switch is not
vacuous. One line per major sweep, which is cadence-limited rather than per page.

Remove once the flake is understood.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 38d9fd2d48

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread vm/ByteCodeTranslator/src/cn1_globals.m
Comment thread vm/ByteCodeTranslator/src/cn1_globals.m
Setting CN1_LOG_PAGE_RELEASE in the workflow env handed it to every test in the
suite, and the suite includes a heap-integrity verifier whose runs are timing
sensitive. Both arm64 arms had passed twice in a row before the tracing went in
and both failed on the push that added it -- once with a genuine dangling
reference and once with the verifier failing to detect its own injected defect.
That is suggestive rather than proven, but the scope was wrong either way: only
BibopPageFloorIntegrationTest wanted the counts.

The switch now lives on that test's own ProcessBuilder, so exactly one child
process gets it and no other test's timing is touched. Verified with
CN1_LOG_PAGE_RELEASE explicitly unset in the environment: the run still emits 51
[MAJOR-SWEEP] lines, so the test really is turning it on rather than inheriting
it, and both it and GcHeapIntegrityIntegrationTest pass locally.

This also gives a clean read on those two arm64 failures. If they persist with
the tracing scoped away they are real and were simply never exercised before --
this workflow is the only place GcHeapIntegrityIntegrationTest runs, so it has
never executed on arm64 Linux until now. If they go away, the tracing perturbed
them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 37656fbaee

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread vm/ByteCodeTranslator/src/cn1_globals.m Outdated
…sume protocol

Three review findings. Two fixed, one declined in code.

The atexit report ranked the live counters and zeroed them as it printed. Exit
handlers do not stop the other threads, so a class could be printed, keep
allocating, and be selected again for a second row, and its bytes and count were
read at different instants and need not have described the same allocations.
Everything is now copied once and the ranking consumes the copy; the size
histogram had the same shape and gets the same treatment.

CN1_BIBOP_HEAP_GROWTH_PERCENT was left defined and unused when the live-set floor
was removed -- cn1BibopTriggerFloor was its only reader. A knob that is
documented, settable and inert is worse than no knob, because a footprint
experiment that sets it reads its result as evidence about a value that never
applied. Removed, along with the proportional-live-set rationale above it, which
described a rule this file no longer implements. The paragraph now says what the
constant actually is: a floor a deployment CHOOSES, defaulting to the old value,
and why it is not inferred from the live set -- because this collector cannot
measure one, which three failed attempts established.

Declined, with the reasoning in the code: an atomic block-check-and-reactivate in
cn1PacingPark. The window is real and the report describes it correctly, but it
is CN1_RESUME_THREAD's window -- the collector sets threadBlockedByGC, waits for
threadActive to clear and never re-validates, and every native boundary in the VM
resumes with the same unsynchronised pair. Closing it in one function would leave
the same window open at thousands of others while making this site disagree with
the protocol the collector implements, which is the shape of the defects this
area has already produced. It needs a collector-side acknowledgement applied
everywhere at once, which is a VM thread-protocol change and not part of a
pacing fix.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d4c53542a0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread vm/ByteCodeTranslator/src/cn1_globals.m Outdated
Comment on lines +12355 to +12358
cn1AllocProfSnapBytes[i] =
atomic_load_explicit(&cn1AllocProfBytes[i], memory_order_relaxed);
cn1AllocProfSnapCount[i] =
atomic_load_explicit(&cn1AllocProfCount[i], memory_order_relaxed);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Synchronize the allocation-profile snapshot

When another mutator allocates during this atexit handler, these independent loads still do not form a consistent snapshot: the reporter can read bytes, the mutator can increment both counters, and then the reporter can read the newer count, producing an incorrect average and potentially ranking classes from different instants. The fresh evidence in this revision is that the replacement snapshot still reads each counter separately while explicitly acknowledging that other threads remain active; quiesce recorders or synchronize/capture each byte-count pair atomically before ranking.

Useful? React with 👍 / 👎.

…ented skew

The profile stopped reporting entirely and this restores it. Bisected rather than
reasoned about, after four wrong theories: the cause is the table size. Four
tables of 65536 eight-byte entries is 2MB of BSS, and in the static musl binary
the report then faulted partway through its first scan, with the VM's SEGV
handler swallowing the fault so the handler returned having printed nothing.
8192 prints, 65536 prints nothing, everything else held equal.

The bound was never what made this safe. The overflow row is: it counts what fell
outside, adds it to the total and names the highest id seen, so a table that is
too small says so rather than silently omitting the hottest class -- which was
the original finding. Raising the bound was belt-and-braces on top of that, and
it cost the whole instrument.

Also reverts the consistent snapshot. It needed three more tables of the same
size, which is the same 2MB again, and quiescing the recorders first needed a
usleep inside an atexit handler -- not async-signal-safe, and it blocked the
handler outright on the SIGTERM path this is always reached by. The race the
review reported is real: a class still allocating can have its bytes and count
read at different instants and can take a second row. That skew is now
documented where the ranking happens, because a report that is a few allocations
stale is a working instrument and one that is exactly consistent and silent is
not. This is the profile that localised the keep-alive buffer copy.

Kept from that round, all cheap and correct: atomic class pointers, the
compare-exchange bucket claim, the atomic fetch-max for the highest id, the
pthread_once filter publication, and the histogram's stack-local snapshot, which
is 48 entries and adds no BSS.

Verified end to end on the backend at each step, not just compiled: 8192 emits
totalBytes and its per-class rows again.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@shai-almog
shai-almog merged commit e4dc53f into master Sep 7, 2026
44 checks passed
@shai-almog
shai-almog deleted the parparvm-parallel-mark-arm64 branch September 7, 2026 05:51
shai-almog added a commit that referenced this pull request Sep 8, 2026
Five accepted review findings.

-DCN1_DISABLE_BIBOP DID NOT LINK. bibopGcEpoch is defined inside cn1_globals.m's
#ifndef CN1_DISABLE_BIBOP block, and the load barrier named it unguarded, so a
supported A/B and fallback configuration failed with an undefined _bibopGcEpoch.
Confirmed by building it rather than by reading guards. The epoch is only a
filter -- dropping it enqueues referents that would have been skipped, which is
more work and never less safety -- so the disabled build compares against a value
the mark word cannot equal and the barrier keeps its single comparison.

DROP RECOVERY DID NOT REACH A FIXPOINT. The same defect already fixed in
sub-pass A, reappearing in the recovery loop added a commit earlier: marking a
retained referent traces it, and an object kept alive only that way can itself
hold references whose mark functions register after the loop has passed them.

THE CAPPED SATB EXIT SKIPPED REFERENCES ENTIRELY. The CN1_SATB_MAX_REOPENS branch
drains twice on its way out, and those drains can newly mark an object whose graph
contains a Reference. It was the one exit that left without a reference pass, so a
reachable reference kept an unmarked referent the sweep then freed.

THE DROP CHECK COULD NOT SEE A DROP THAT HAD NOT HAPPENED YET. A get() starting
after the pre-loop check loads an unmarked referent, and a failed enqueue moves
the counter only once clearing has already decided it was safe. The pass now
records what each entry cleared, quiesces -- every getter registers for the
duration of its load, so an in-flight count of zero means every getter that
overlapped has finished and published its drop -- and re-reads the counter,
marking what it cleared if it moved. The stores are not undone and need not be: a
cleared reference answering null is legal, the object being freed under a mutator
is not.

THE EMERGENCY PATH HAD NOWHERE TO RECORD. It clears at discovery precisely
because the list could not grow, so the recovery above had nothing to consult and
a racing get() with a failed enqueue would have been handed a freed pointer. It
now remembers what it cleared in a fixed preallocated array -- allocation being
the one thing unavailable there -- and REFUSES TO CLEAR when that is full,
marking instead: memory the emergency wanted back is retained, which is worse
than clearing and far better than a dangling read.

Rebased onto master, which landed the parallel-mark collector work (#5717) in the
same two files; the conflict was additive on both sides and the resolved tree was
compiled before the rebase continued.

Gates: run-gc-verify.sh green with all three self-tests firing, run-gauntlet.sh
green with all eleven tortures bit-identical and both GC stop modes, and all five
build shapes compile -- plain, CN1_GC_CONFORM, CN1_DISABLE_BIBOP,
CN1_DISABLE_SATB and CN1_GC_VERIFY.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
shai-almog added a commit that referenced this pull request Sep 8, 2026
Five accepted review findings.

-DCN1_DISABLE_BIBOP DID NOT LINK. bibopGcEpoch is defined inside cn1_globals.m's
#ifndef CN1_DISABLE_BIBOP block, and the load barrier named it unguarded, so a
supported A/B and fallback configuration failed with an undefined _bibopGcEpoch.
Confirmed by building it rather than by reading guards. The epoch is only a
filter -- dropping it enqueues referents that would have been skipped, which is
more work and never less safety -- so the disabled build compares against a value
the mark word cannot equal and the barrier keeps its single comparison.

DROP RECOVERY DID NOT REACH A FIXPOINT. The same defect already fixed in
sub-pass A, reappearing in the recovery loop added a commit earlier: marking a
retained referent traces it, and an object kept alive only that way can itself
hold references whose mark functions register after the loop has passed them.

THE CAPPED SATB EXIT SKIPPED REFERENCES ENTIRELY. The CN1_SATB_MAX_REOPENS branch
drains twice on its way out, and those drains can newly mark an object whose graph
contains a Reference. It was the one exit that left without a reference pass, so a
reachable reference kept an unmarked referent the sweep then freed.

THE DROP CHECK COULD NOT SEE A DROP THAT HAD NOT HAPPENED YET. A get() starting
after the pre-loop check loads an unmarked referent, and a failed enqueue moves
the counter only once clearing has already decided it was safe. The pass now
records what each entry cleared, quiesces -- every getter registers for the
duration of its load, so an in-flight count of zero means every getter that
overlapped has finished and published its drop -- and re-reads the counter,
marking what it cleared if it moved. The stores are not undone and need not be: a
cleared reference answering null is legal, the object being freed under a mutator
is not.

THE EMERGENCY PATH HAD NOWHERE TO RECORD. It clears at discovery precisely
because the list could not grow, so the recovery above had nothing to consult and
a racing get() with a failed enqueue would have been handed a freed pointer. It
now remembers what it cleared in a fixed preallocated array -- allocation being
the one thing unavailable there -- and REFUSES TO CLEAR when that is full,
marking instead: memory the emergency wanted back is retained, which is worse
than clearing and far better than a dangling read.

Rebased onto master, which landed the parallel-mark collector work (#5717) in the
same two files; the conflict was additive on both sides and the resolved tree was
compiled before the rebase continued.

Gates: run-gc-verify.sh green with all three self-tests firing, run-gauntlet.sh
green with all eleven tortures bit-identical and both GC stop modes, and all five
build shapes compile -- plain, CN1_GC_CONFORM, CN1_DISABLE_BIBOP,
CN1_DISABLE_SATB and CN1_GC_VERIFY.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
shai-almog added a commit that referenced this pull request Sep 9, 2026
* Give ParparVM real weak and soft references

The collector had no notion of a weak root. java.lang.ref.WeakReference held
its referent in an ordinary field, so the translator emitted a gcMarkObject for
it and the referent was STRONG -- a "weak" reference pinned its referent for the
life of the process, and every cache built on
CodenameOneImplementation.createSoftWeakRef (the EncodedImage decode cache,
Image's scale and RGB caches, Border's round-rect cache, rasterised gradients)
was unbounded. There was no SoftReference at all.

The referent now lives in Reference and the translator does not trace it
(ByteCodeClass.isReferenceReferent): it emits cn1GcDiscoverReference, which
hands the collector the field addresses and decides soft retention on the spot.
Clearing happens in cn1GcProcessReferences, using the sweep's own liveness test
-- both halves of the sweep free on `mark != -1 && mark < currentGcMarkValue - 1`
and nothing else may be cleared, because a dangling read on this VM is a native
crash no Java catch can see.

Three things about the placement, none of them optional:

- The clear pass runs INSIDE the SATB termination loop, barrier still armed. A
  thread scanned and released early can pull a referent out through get() and
  hold it in a local the collector has walked past, and that referent is then
  neither marked nor fresh -- the one case the sweep's "already marked or FRESH"
  invariant does not cover. get() carries a load barrier, so a racing read makes
  the trial clear of gcSatbActive find a non-empty log, which re-arms and re-runs
  the fixpoint and this pass with it.

- That barrier is FILTERED (CN1_SATB_REF_LOAD). Logging every referent read is
  not a cost but a failure: cn1SatbEnqueue takes a mutex per accepted reference
  and get() on a hot cache is called far more often than any store barrier sees.
  Unfiltered it put over 10,000 entries in the log per cycle and reached
  CN1_SATB_MAX_REOPENS on EVERY cycle. Skipping referents already marked this
  epoch or fresh -- exactly the ones the clear pass would refuse to clear -- took
  passes 32 -> 1 and refMs 0.06 -> 0.005.

- Soft retention is ranked by age since the last get(), decided when the mark
  first reaches the reference so the mark stays single pass. Deciding afterwards
  would mean a second reachability closure over the retained set, on a mark that
  already spends most of its time in the grace pass.

Measured (RefPolicy + ab-refs.sh, five interleaved reps, checksums identical
across every arm). References themselves are unambiguous: 255/256 unreachable
referents reclaimed against 0/256, for 1.8-4% of mark time, with a vm/benchmarks
geomean of 1.011 over 12 interleaved reps against master. The RANKING is a cheap
rider rather than the payoff -- against never-clearing it is the same hit rate
for about a megabyte. What it does beat decisively is the pressure-triggered
alternative, which gave up 12 points of hit rate for zero footprint saving and
was worse on both axes at a 160MB ceiling; that arm is the model of the iOS
port's didReceiveMemoryWarning -> flushSoftRefMap.

vm/CLAUDE.md carries the table and the three conclusions that were drawn from
single runs and turned out to be wrong, including the one that generalises: a
pressure-triggered cache policy cannot work on this collector, because the pacing
loop defends the reserve by throttling the mutator, so headroom converges on any
threshold placed there and never crosses it.

This change is confined to the VM. The porting layer, the iOS softReferenceMap
override and the core call sites are deliberately untouched and follow separately,
since they change behaviour in every iOS app and want their own bisect point.

Gates: run-gc-verify.sh green (RefPolicy clean over 47 verify passes, both
fault-injection self-tests firing), run-gauntlet.sh bit-identical to the host
JVM, vm/tests 552 tests / 0 failures, GC integration leg 10/10 including
GcSteadyStateIntegrationTest and ProcessBudgetPacingIntegrationTest.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Make reference field access atomic, and decide aliases together

Both from review of the previous commit, and both premises hold.

ATOMICITY. The collector cleared the referent with a plain store while the
generated accessor read it with a plain load, on threads that run concurrently
by design -- a data race, undefined in C however benign the emitted instruction
is on the targets built here. It was also an inconsistency within the same
function: the line below the clear already read the MARK WORD with an acquire
atomic, because cn1_globals.m converted its hand-written stores for exactly this
reason. Relaxed atomics now at every site that touches the referent: the
generated getter and setter (so Reference.clear() and the constructor are
covered), CN1_SATB_REF_LOAD, and the three collector paths. cn1TouchAge gets the
same treatment -- the mutator stamps it from the accessor while the collector
reads and ages it, which is the identical defect one field over.

ALIASES. Two references to one referent were decided at different instants, so a
get() landing mid-pass could stamp the second as recently read after the first
had been cleared -- one alias answering null while another answers the object.
The contract says all references to a weakly reachable object are cleared
atomically. For a cache a split is a spurious miss; for the callers that use a
reference as a LIFETIME ORACLE, reading a null get() as proof the referent died,
it is a false death report on one alias while the object is alive through
another. That is the failure mode that makes the iOS soft-reference table
dangerous today.

The clear pass is now two sub-passes: A marks every referent read this cycle and
drains, B then clears on the referent's mark word alone. Liveness is a property
of the referent, so every alias reads the same answer and they are cleared
together or kept together. This removes the possibility rather than narrowing the
window -- a get() racing sub-pass B still gets a non-null referent and still
enqueues it, so the object survives and every alias is cleared, which is a legal
spurious clear and is what "atomically" asks for.

RefPolicy grows an alias phase with a concurrent reader. Read the comment on it
before citing it: it is NOT a self-test. Built with the new
-DCN1_REF_NO_ALIAS_ATOMICITY arm, which restores the single-loop form that has
the bug, it still reports ALIAS_SPLIT=0/256 -- the window is microseconds and
could not be opened from a driver. Two earlier versions of that phase were worse
and are documented so they are not rebuilt: one read the aliases only after
quiescing, where every alias carries the same stamp and nothing can fail; the
other kept every referent marked, so ALIAS_CLEARED_GROUPS was 0/256 and the thing
being tested never happened. The fix ships because the two-sub-pass form is
unconditionally correct and simpler, not because a test proved the old one broken
-- the same footing as CN1_NO_BULK_INSERTION_BARRIER.

Gates: run-gc-verify.sh green (RefPolicy clean over 71 verify passes, both
fault-injection self-tests firing), run-gauntlet.sh green with all eleven
tortures bit-identical and both GC stress modes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Release soft referents on allocation failure, and stop losing touches

Both from review, and both premises hold.

ALLOCATION FAILURE. The retention ladder reads cn1ProcessHeadroom(), which
answers -1 wherever there is no per-process budget to probe -- every desktop
build and the simulator. There it could only ever say "plenty", so a soft
referent read at least once every CN1_REF_SOFT_RETAIN_MAX cycles was never
dropped however tight memory actually was, while codenameOneGcMalloc's failure
path only asked for a collection and retried. That breaks the one guarantee
SoftReference makes -- every soft reference cleared before the VM gives up -- and
turns recoverable pressure into a retry loop that collects nothing.

cn1RefDropAllSoftReferents() raises a latch on the failure path which the next
cycle consumes, overriding every policy including "never clear". A latch rather
than a direct write, because cn1RefBeginCycle recomputes the budget at the top of
each cycle and would erase one.

Proven, not argued: with failures injected while soft referents are live, the
emergency cycles report retained=0 and cleared=1097 against retained>0 on the
same workload uninjected. Getting that proof needed CN1_SIMULATE_ALLOC_FAILURES
to grow a "<n>:<skip>" form -- it could only fail the FIRST n allocations, which
is startup, and a state that exists only at startup cannot exercise anything the
program builds later. Every attempt without it produced real emergency cycles
that all reported discovered=0.

LOST TOUCHES. Ageing was a load, a compute and a store, so a get() landing
between the load and the store was erased outright -- the referent looked cold
with no record it had been read at all, and the age was not even reset. Worse, it
falsified a claim: consuming CN1_REF_TOUCHED at DISCOVERY meant the clear pass's
"was it read?" fallback only ever covered reads landing after discovery, not the
whole cycle as its comment implied.

The ageing therefore moves out of discovery to the end of the clear pass, and is
a compare-exchange. Any read anywhere in the cycle is now still visible to
sub-pass A, so the fallback covers what it says it does, and a racing touch can
no longer be lost -- a failed exchange reloads, sees CN1_REF_TOUCHED and resets
the age to 0, which is what the touch means.

Two consequences worth naming. The CN1_REF_NO_ALIAS_ATOMICITY arm had to start
ageing inline, or with discovery no longer doing it the arm would silently have
become "retain everything" rather than the shape it exists to reproduce. And
Reference.cn1TouchAge now starts at 0 rather than TOUCHED: TOUCHED means "read
since the collector last aged this", which is false for a reference nobody has
called get() on, and starting there made the clear pass mark the referent of
every newly discovered reference for a cycle, weak ones included.

Gates: run-gc-verify.sh green (RefPolicy clean over 71 verify passes, both
fault-injection self-tests firing), run-gauntlet.sh green with all eleven
tortures bit-identical and both GC stop modes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Retain a referent discovery could not record, and load it once

Two accepted review findings and one declined, plus a driver that had gone
vacuous again.

A DROPPED DISCOVERY MUST RETAIN. When the discovery list could not grow -- the
realloc failed, or growth was declined because a thread is signal-frozen -- the
entry was dropped AND the referent left unmarked. The comment above it claimed
that was safe because "the referent stays reachable through a field nothing
cleared", which is exactly backwards and is the kind of sentence that reads as
obviously true: nothing else marks a weak referent, that being the point of a
weak edge, so the sweep frees it and the uncleared field becomes a dangling
pointer inside a perfectly reachable Reference, handed to the next get(). On this
VM that is a native crash no Java catch can see, on a path only ever taken when
memory is already short. The drop path now marks the referent, which costs one
deferred reclaim.

ONE LOAD IN THE ACCESSOR. CN1_SATB_REF_LOAD only read the field INSIDE its
gcSatbActive branch, so with the barrier down the accessor loaded again
afterwards and the gap between the two was a hole: a thread that read the flag as
0, was SIGUSR2-frozen with the referent not yet in any register, scanned,
released, and only then loaded, came away holding an unmarked referent nothing
had enqueued. The accessor now loads once into a local and the barrier
(CN1_SATB_REF_KEEP) acts on that same value, so what get() returns is what the
barrier saw -- and with the load first the value is in a register before any
freeze, where the conservative root scan finds it.

ATOMIC PUBLICATION OF ALIAS CLEARING IS DECLINED, and the reasoning is in the
code at the store rather than only here. Making N stores visible as one step
needs a lock that Reference.get() also takes, and get() is the single hot path
this design exists to keep free of one; HotSpot does not do it either, clearing
referents one at a time without synchronising against get(). The contract's
requirement is that the DECISION covers every alias together, which the two
sub-passes provide and which was the part genuinely broken before them. The
residual window is transient and self-healing -- a get() inside it arms the
barrier and resurrects the object.

The alias phase had gone hollow again after ageing moved to the clear pass: with
the stamp surviving the whole cycle, a reader touching one alias of EVERY group
kept every group alive and ALIAS_CLEARED_GROUPS read 0/256, which is the vacuum
that phase exists to detect. It now reads one group in four, and collection is
back to 191/256.

Gates: run-gc-verify.sh green (RefPolicy clean over 71 verify passes, both
fault-injection self-tests firing), run-gauntlet.sh green with all eleven
tortures bit-identical and both GC stop modes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Process references to a fixpoint, and register the get() barrier

Two accepted review findings, both real, both about the same thing: the
reference pass assumed it could see the whole picture at one instant.

DISCOVERY IS NOT DONE WHEN THE PASS STARTS. cn1GcProcessReferences snapshotted
cn1RefDiscoveredTop once and cleared against that snapshot -- but sub-pass A
DRAINS, and a drain discovers references. Marking a touched referent traces it,
and an object kept alive only by that reference can itself hold weak references
whose mark functions call cn1GcDiscoverReference; those landed past the snapshot
and sub-pass B never looked at them. The result is the failure this whole design
exists to prevent: a reachable Reference nothing cleared, holding a referent the
sweep freed. Sub-pass A now repeats until a drain adds nothing new, which
terminates because the set only grows and is bounded by the live set.

THE BARRIER HAD TO JOIN THE TERMINATION HANDSHAKE. Checking gcSatbActive and then
calling cn1SatbEnqueue is not enough here: the enqueue takes a mutex, so a thread
can pass the check, be delayed acquiring it, and land its entry in a log the
collector has already stopped draining -- after which the referent it is about to
return gets swept. The per-store barrier accepts precisely that window, and the
argument it accepts it on does NOT extend to this path: "a reference stored after
the fixpoint is already marked or FRESH" is true of a store and false of a weak
referent handed out by get(), which is neither. cn1SatbBulkBegin already registers
before it answers, and gcSatbTerminating stays raised across the whole termination
loop including reference processing, so reusing that handshake closes it with the
machinery already present. Off-GC the fast path is unchanged: two
predicted-not-taken flag loads.

Also restores the CN1_REF_NO_ALIAS_ATOMICITY arm, which the sub-pass rewrite had
silently taken with it -- RefPolicy's alias phase documents that flag, so losing
it would have left a comment describing a build that no longer existed.

Gates: run-gc-verify.sh green (RefPolicy clean over 71 verify passes, both
fault-injection self-tests firing), run-gauntlet.sh green with all eleven
tortures bit-identical and both GC stop modes. The emergency soft-drop still
fires under injected allocation failure (retained=0 at softBudget=-1).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Let the emergency clear soft referents without the discovery list

Accepted review finding, and it is two of this branch's own fixes colliding at
the one moment both were written for.

The emergency retention budget is raised by an allocation FAILURE. An allocation
failure is also the likeliest reason the discovery list's realloc cannot grow. So
the conservative fallback added for weak-reference safety -- mark anything that
could not be recorded -- was marking exactly the soft referents the emergency had
just condemned, the collection freed nothing, and codenameOneGcMalloc's retry
loop had nothing to make progress against. Each fix is right on its own; together
they livelock.

A soft reference the emergency has condemned needs no list entry, because the
decision is already final rather than deferred. Its field is cleared in place,
which is the allocation-free path this situation calls for, and it is safe for
the same reason the ordinary clear is: a get() that already loaded the referent
enqueued it through the armed barrier and keeps it alive for the cycle, and a
get() after the store reads null.

Weak references, and anything read since the last ageing, still take the marking
branch -- that is where a dangling pointer would actually come from, since a
mutator may be holding the referent in a local the collector has walked past.

The accepted cost is stated at the code: two aliases of one SOFT referent can
disagree when only some of them were recorded. That is confined to soft
references, which are caches by definition, and the alternative is an allocator
that cannot make progress. The callers that read a null get() as proof of death
use weak references, which mark.

Gates: run-gc-verify.sh green (RefPolicy clean, both fault-injection self-tests
firing), run-gauntlet.sh green with all eleven tortures bit-identical and both GC
stop modes, and the emergency drop still fires under injected allocation failure.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Hold the SATB registration across the referent load

Two accepted review findings. The P1 is this branch's own optimisation undoing
its own fix, which is worth naming plainly.

REGISTRATION MUST SPAN THE LOAD. The previous commit made the get() barrier
register through cn1SatbBulkBegin, whose entire value is that it registers FIRST
and reports whether logging is needed afterwards -- and then gated entry to it on
an outer read of the same two flags, as a fast path. That puts the race back one
level out: a thread can be descheduled between the outer read and the
registration, and the collector can clear the field, finish its empty final take,
lower gcSatbTerminating and quiesce with the in-flight count still zero. The
enqueue is then declined and the sweep frees the referent get() is about to
return. Nothing may be sampled before the registration, so the accessor now
brackets the whole load: CN1_REF_LOAD_BEGIN, load, CN1_SATB_REF_KEEP,
CN1_REF_LOAD_END.

The cost is two seq_cst read-modify-writes on every Reference.get(), paid
unconditionally, and there is no sound way to skip them: any flag consulted
before registering can go stale in exactly the window the registration exists to
close. With it held across the load a false answer is safe rather than merely
unlikely -- the collector cannot be mid-termination because its quiesce waits for
this registration, so either no mark is running and a later one scans this thread
with the value already in a register, or reference processing is complete and a
field still holding a pointer was not condemned.

THE OOM FALLBACK MUST NOT CLEAR A STRONGLY REACHABLE REFERENT. The emergency
clear added last commit did not consult the mark word, so an application holding
both an ordinary field and a SoftReference to one object could watch get() answer
null under allocation pressure for an object that was never softly reachable. It
now skips anything already marked. Partial by construction, and said so at the
code: the mark is still running there, so a referent a strong edge reaches LATER
in the cycle is not yet marked and can still be cleared. Being certain would mean
deferring to the clear pass, which is precisely what that path exists because it
cannot do.

Gates: run-gc-verify.sh green (RefPolicy clean over 69 verify passes, both
fault-injection self-tests firing), run-gauntlet.sh green with all eleven
tortures bit-identical and both GC stop modes. Both the -DCN1_DISABLE_SATB and
default arms build and run; an earlier revision of this commit left the header's
conditional nesting unbalanced, which is why that arm is now checked explicitly.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Finish the referent's atomic accesses, and stop clearing after a dropped log entry

Two accepted review findings, both leftovers of earlier fixes on this branch
rather than new ground.

HALF A RACE WAS FIXED. Making the referent's accesses atomic converted the getter
and the setter's own store but left CN1_SATB_DELETE, emitted immediately before
that store, reading the same field through a plain JAVA_OBJECT volatile*. The
collector writes JAVA_NULL there atomically from cn1GcProcessReferences, so the
pair stayed a mixed atomic/non-atomic access -- the exact defect the earlier
change set out to remove. CN1_SATB_DELETE_REF is the deletion barrier with an
atomic load, used for this field only; every other field keeps the generic macro,
which is correct for them because nothing else writes them concurrently.

A DROPPED LOG ENTRY VOIDS THE CLEAR PASS'S EVIDENCE. cn1SatbEnqueue discards a
reference when its stack cannot grow. The comment there argues that is
survivable, and for an ordinary STORE it is -- "only re-opens the original race".
It is not survivable for a referent Reference.get() has already handed to a
mutator: the enqueue was the only record that it escaped, sub-pass B decides
purely on mark state, and the final take stays empty so nothing re-opens, leaving
the sweep free to reclaim an object a thread is holding. Registering the load in
the termination handshake does not help here; it delays the take, it does not
stop a drop.

Nothing on that path can allocate its way out, so the pass now snapshots the drop
count at cycle start and declines to clear at all if it moved. That costs one
cycle of reclaim in a process that is already out of memory, and it deliberately
does not disable the emergency soft-drop, which clears at DISCOVERY and never
touches the log.

Gates: run-gc-verify.sh green (RefPolicy clean over 71 verify passes, both
fault-injection self-tests firing), run-gauntlet.sh green with all eleven
tortures bit-identical and both GC stop modes; the emergency drop still fires
under injected allocation failure and the driver's weak, alias and cache
assertions are unchanged.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Count take-side SATB losses, and stop clearing when SATB is compiled out

Two accepted review findings, and one pre-existing bug found next to the first.

THE DROP COUNTER GUARDED HALF THE LOSS PATH. The previous commit invalidated
reference clearing when cn1SatbEnqueue discarded an entry, but cn1SatbTake
discards too: it resets gcSatbTop unconditionally and, when its scratch buffer
cannot be grown, reports the batch as EMPTY. Entries that were logged
successfully are then thrown away and the collector reads "nothing slipped in",
which is exactly the signal that lets termination finish. To the referent that
gets swept a lost enqueue and a lost batch are the same event, so the take now
counts as a drop as well.

PRE-EXISTING, in the same function and worth its own paragraph: realloc's result
was assigned straight back over scratch, so a failure lost the buffer that was
already there, and scratchCap was advanced whether or not the growth succeeded.
After a single failure every later take saw n <= scratchCap, skipped the realloc,
found scratch NULL and returned 0 -- the barrier logging into a stack nothing
would ever drain again, permanently and silently. It now grows through a
temporary and advances the cap only on success.

-DCN1_DISABLE_SATB WAS UNSOUND, NOT MERELY SLOWER. That arm compiles out the load
barrier that makes handing a weak referent to a mutator safe, but reference
processing is not part of SATB and kept running: a thread released after its stack
scan could load a referent, have the field cleared underneath it and the object
swept before it could use the pointer. Reference processing now switches off with
the barrier, degrading to the behaviour that preceded this feature -- referents
traced strongly and never cleared -- which is the right fallback for an escape
hatch and keeps the arm measuring the barrier's cost rather than a different
collector. Verified rather than assumed: WEAK_DEAD_CLEARED reads 0/256 in that
arm against 255/256 by default.

Gates: run-gc-verify.sh green (RefPolicy clean over 70 verify passes, both
fault-injection self-tests firing), run-gauntlet.sh green with all eleven
tortures bit-identical and both GC stop modes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Stop the verifier being blind to referents, and close a heap overflow

Two accepted review findings. One is a heap overflow this branch introduced an
hour ago; the other says the gate that was supposed to catch such things could
not see the field in question at all.

THE OVERFLOW WAS MINE. Growing cn1SatbTake's scratch buffer through a temporary
keeps the old allocation when realloc fails -- which was the point, the previous
shape leaked it -- and therefore leaves scratch NON-NULL and SMALLER than n. The
guard still tested only scratch != 0, so the memcpy wrote n entries into an
allocation sized for fewer: heap corruption written by the collector under memory
pressure, strictly worse than the leak it replaced. The buffer is usable only if
it exists AND scratchCap >= n.

THE VERIFIER COULD NOT SEE THE REFERENT. cn1GcVerifyHeap walks survivors through
the generated mark functions and relies on every reference field reaching
gcMarkObject, whose verify branch classifies it. Suppressing that call for the
referent -- the very thing that makes the edge weak -- also took the referent out
of the verifier's reach, so a live Reference holding a pointer into reclaimed
memory passed with violations=0. Every "clean over N verify passes" recorded on
this branch before now was silent about the referent specifically, which is the
one thing this work risks. cn1GcDiscoverReference now routes it to
cn1GcVerifyChild, ahead of the ageing and the dedupe: a verify walk is not a
collection cycle, and letting it age references or consume the dedupe stamp would
corrupt what the next real cycle reads and drop exactly the repeat visits a
whole-heap walk produces.

PROVEN, not assumed. CN1_GC_FAULT=refnoclear leaves a dead referent in its field
instead of clearing it, and the verifier reports DANGLING REFERENCE; clean it
reports none. run-gc-verify.sh gains self-test3 so this cannot go quietly blind
again. Note the obvious-looking fault is the wrong one and was tried first:
clearing MORE references than liveness warrants only produces extra nulls, which
are safe, and it reported violations=0 -- stopping there would have "confirmed"
the hook while proving nothing. The dangling direction is clearing LESS.

The fault's use is inside CN1_GC_VERIFY because the cn1GcFault* family is
declared there. Unguarded it broke every ordinary build while the verifier build
-- the one configuration in which the symbol exists -- kept passing, so all three
shapes are now built explicitly: plain, CN1_GC_CONFORM and CN1_GC_VERIFY.

Gates: run-gc-verify.sh green with all three self-tests firing, run-gauntlet.sh
green with all eleven tortures bit-identical and both GC stop modes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Retain discovered referents when a dropped log abandons the clear pass

Accepted review finding, and a repeat of a mistake this branch already corrected
once. The drop-counter fallback returned without clearing, which leaves every
discovered referent unmarked AND its field non-null -- so the sweep frees objects
that live References still point at. That is the dangling read the pass exists to
prevent, produced by the code meant to prevent it.

"Skip the clear" is not "keep the referent alive". Nothing else marks a weak
referent; that is what makes the edge weak. The note on the unrecorded-discovery
path says exactly this, and the same confusion reappeared here.

The fallback now marks every discovered referent and drains before returning.

Gates: run-gc-verify.sh green with all three self-tests firing, run-gauntlet.sh
green with all eleven tortures bit-identical and both GC stop modes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Measure what the Reference.get() load barrier actually costs

The barrier registers in the SATB termination handshake, which is two seq_cst
read-modify-writes on every Reference.get() and cannot be skipped soundly -- any
flag sampled before registering can go stale in the window the registration
exists to close. Whether that price is worth paying is a question for a number,
not an intuition, so this adds the means to get one.

-DCN1_REF_NO_LOAD_BARRIER compiles the registration and the enqueue out, leaving
the load and the touch stamp. It is UNSOUND and is not a shipping configuration;
it is the arm the cost is measured against.

cn1RefGets counts get() calls, and [GCREF-TOTAL] prints the run total at exit.
The per-cycle [GCREF] line cannot answer "how often does this workload call
get()", because it only prints when a collection happens -- so a get()-heavy but
allocation-light phase, which is exactly the shape that costs the barrier most,
leaves the last line stranded early in the run. Costing the barrier off that
number understates the calls and overstates the nanoseconds: it read 153,408 for
a run that made 6,155,728.

Measured on this host, RefPolicy, nine interleaved reps, both arms
-DCN1_GC_CONFORM -flto=thin:

  get-dominated   64 keys x 64B, 6M reads, no churn   6,155,728 gets
      +0.07% median, +0.35% floor  ->  +0.45 ns median, +2.14 ns floor per get()

  image-cache     512 keys x 32KB, 400k reads, churn    556,416 gets
      +0.10% median, -0.30% floor  ->  unmeasurable at this call volume

So the barrier costs roughly half a nanosecond to two nanoseconds per call, and
0.35% of wall time only at 1.6 MILLION get() calls per second. An image-heavy
screen redrawing fifty encoded images at 60fps calls it three thousand times a
second, some three orders of magnitude below the rate at which it becomes
visible.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Close the remaining reference-clearing races, and fix the no-BiBOP build

Five accepted review findings.

-DCN1_DISABLE_BIBOP DID NOT LINK. bibopGcEpoch is defined inside cn1_globals.m's
#ifndef CN1_DISABLE_BIBOP block, and the load barrier named it unguarded, so a
supported A/B and fallback configuration failed with an undefined _bibopGcEpoch.
Confirmed by building it rather than by reading guards. The epoch is only a
filter -- dropping it enqueues referents that would have been skipped, which is
more work and never less safety -- so the disabled build compares against a value
the mark word cannot equal and the barrier keeps its single comparison.

DROP RECOVERY DID NOT REACH A FIXPOINT. The same defect already fixed in
sub-pass A, reappearing in the recovery loop added a commit earlier: marking a
retained referent traces it, and an object kept alive only that way can itself
hold references whose mark functions register after the loop has passed them.

THE CAPPED SATB EXIT SKIPPED REFERENCES ENTIRELY. The CN1_SATB_MAX_REOPENS branch
drains twice on its way out, and those drains can newly mark an object whose graph
contains a Reference. It was the one exit that left without a reference pass, so a
reachable reference kept an unmarked referent the sweep then freed.

THE DROP CHECK COULD NOT SEE A DROP THAT HAD NOT HAPPENED YET. A get() starting
after the pre-loop check loads an unmarked referent, and a failed enqueue moves
the counter only once clearing has already decided it was safe. The pass now
records what each entry cleared, quiesces -- every getter registers for the
duration of its load, so an in-flight count of zero means every getter that
overlapped has finished and published its drop -- and re-reads the counter,
marking what it cleared if it moved. The stores are not undone and need not be: a
cleared reference answering null is legal, the object being freed under a mutator
is not.

THE EMERGENCY PATH HAD NOWHERE TO RECORD. It clears at discovery precisely
because the list could not grow, so the recovery above had nothing to consult and
a racing get() with a failed enqueue would have been handed a freed pointer. It
now remembers what it cleared in a fixed preallocated array -- allocation being
the one thing unavailable there -- and REFUSES TO CLEAR when that is full,
marking instead: memory the emergency wanted back is retained, which is worse
than clearing and far better than a dangling read.

Rebased onto master, which landed the parallel-mark collector work (#5717) in the
same two files; the conflict was additive on both sides and the resolved tree was
compiled before the rebase continued.

Gates: run-gc-verify.sh green with all three self-tests firing, run-gauntlet.sh
green with all eleven tortures bit-identical and both GC stop modes, and all five
build shapes compile -- plain, CN1_GC_CONFORM, CN1_DISABLE_BIBOP,
CN1_DISABLE_SATB and CN1_GC_VERIFY.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Define the GC epoch mirror unconditionally instead of working around it

The load barrier needed bibopGcEpoch, which was defined inside cn1_globals.m's
#ifndef CN1_DISABLE_BIBOP block, so that configuration failed to link. The first
fix taught the barrier to compare against a sentinel when BiBOP was compiled out
-- a configuration branch on a hot path to accommodate a symbol that had no
reason to be conditional.

The epoch is a plain mirror of currentGcMarkValue for mutator-side reads. Nothing
about it belongs to the page heap; it sat inside that guard by accident of
placement. Hoisting it out removes the macro entirely and returns the barrier to
one comparison with no configuration in it.

Keeping the arm alive rather than dropping it costs nothing now and keeps
vm/CLAUDE.md honest: -DCN1_DISABLE_BIBOP is listed there as an ablation, and an
ablation that does not link is a trap for whoever reaches for it. With BiBOP off
nothing advances the epoch, so the barrier enqueues referents it would otherwise
have skipped -- the epoch is a filter, and a stale one only ever declines to
skip.

Six build shapes verified: plain, CN1_GC_CONFORM, CN1_DISABLE_BIBOP,
CN1_DISABLE_SATB, CN1_GC_VERIFY, and CN1_DISABLE_BIBOP with CN1_GC_CONFORM
together.

Gates: run-gc-verify.sh green with all three self-tests firing, run-gauntlet.sh
green with all eleven tortures bit-identical and both GC stop modes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Serialize emergency recovery slots, and stop the epoch mirror going stale

Four accepted review findings; three are defects in the recovery paths added a
commit earlier.

EMERGENCY SLOT RESERVATION WAS RACY. cn1RefEmergencyTop++ runs after cn1RefMutex
has been released, and mark functions run on however many workers
gcMarkDrainParallel is using -- so increments were lost and two workers could
publish into one slot. An emergency clear that goes unrecorded is exactly the
case the drop recovery cannot repair, leaving the sweep free to take the referent
under a racing get(). The index is now an atomic reserve-then-publish, and a
reservation that cannot be satisfied means the clear does not happen at all.

THE EARLY DROP FALLBACK DID NOT RETAIN EMERGENCY CLEARS. It returns before the
post-clear recovery, and a referent cleared by the emergency path is gone from
its field -- it exists only in the recovery array. Retaining cn1RefDiscovered
alone therefore left those objects to be swept under a getter that had already
been handed one.

THE POST-CLEAR RECOVERY DRAINED ONCE. The same fixpoint defect as the other two
recovery loops: a referent brought back can hold a further Reference whose mark
function registers during the drain, after the loops have run. It now iterates,
and retains anything the drain discovered rather than clearing it -- by that
point the pass is past where clearing is safe.

THE STALE EPOCH WAS NOT HARMLESS. The previous commit hoisted bibopGcEpoch out of
the BiBOP guard so -DCN1_DISABLE_BIBOP would link, and its comment claimed a
frozen mirror cost only some extra enqueues. It costs convergence:
CN1_SATB_REF_KEEP skips a referent whose mark equals the epoch, and against a
mirror frozen at 1 that matches nothing from the second collection onward -- the
unfiltered shape already measured on this branch to put over 10,000 entries a
cycle into the log and reach CN1_SATB_MAX_REOPENS every cycle. A filter that
silently stops filtering is a cliff, not a rounding error.

The epoch is now published by codenameOneGCMark, which every cycle passes through
whether or not the page heap is compiled in, leaving one writer instead of two.
Verified rather than argued: with -DCN1_DISABLE_BIBOP -DCN1_GC_CONFORM the
termination loop reports passes=1 on all 32 cycles.

Gates: run-gc-verify.sh green with all three self-tests firing, run-gauntlet.sh
green with all eleven tortures bit-identical and both GC stop modes; plain,
CN1_GC_CONFORM, CN1_DISABLE_BIBOP, CN1_DISABLE_SATB and CN1_GC_VERIFY all build.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Retain rather than clear on the capped SATB termination path

Accepted review finding, and the defect was in a comment before it was in the
code.

The capped path -- reached when a mutator storms the barrier past
CN1_SATB_MAX_REOPENS -- called cn1GcProcessReferences on the way out, justified
by "the barrier is already down on this path, so a get() racing this pass cannot
log". That sentence is false. cn1SatbBulkBegin answers gcSatbActive OR
gcSatbTerminating, and gcSatbTerminating stays raised until after the loop, so a
getter there registers, enqueues successfully, and lands its entry in a log this
path never takes again. Clearing on that basis can free an object a getter is in
the middle of being handed.

Rather than correct the reasoning and keep clearing, the path now RETAINS.
Nothing is cleared, so nothing can dangle however the race falls, and no argument
about flag ordering is load-bearing. The cost is one cycle of reclaim on a path
whose own comment records reaching it 0-4 times against a cap of 32.

cn1GcRetainAllReferences factors out the retain-to-fixpoint walk that the drop
fallback and this path both need -- written three times by hand across this
branch, and the fixpoint was missing from two of them. It covers discovered
referents, referents already cleared this cycle, and the emergency array, and
iterates because marking a retained referent can reach a Reference whose mark
function registers during the drain.

Gates: run-gc-verify.sh green with all three self-tests firing, run-gauntlet.sh
green with all eleven tortures bit-identical and both GC stop modes; plain,
CN1_GC_CONFORM, CN1_DISABLE_BIBOP, CN1_DISABLE_SATB and CN1_GC_VERIFY all build.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Recover references when the final SATB take loses its batch

Accepted review finding, and the last unguarded corner of the same take-side
loss.

cn1SatbTake reports an empty batch for two different reasons: the log really was
empty, or its scratch buffer could not grow and the entries were thrown away. It
records the second case in cn1SatbDrops -- but the final catch in
codenameOneGCMark runs AFTER cn1GcProcessReferences made its last drop check, and
`if(n == 0) break` reads an empty batch as "the mark is closed". Nothing looked at
the new counter value, so a Reference.get() that logged its referent
successfully, and then had that batch discarded here, kept a pointer the
following sweep freed.

The break now retains when the counter has moved since the cycle began. Retaining
rather than another clear pass, because the barrier is coming down at that point
and there is no sound basis left for calling anything dead.

Gates: run-gc-verify.sh green with all three self-tests firing, run-gauntlet.sh
green with all eleven tortures bit-identical and both GC stop modes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Re-arm the SATB barrier before tracing recovered referents

Accepted review finding, and it catches this change breaking a rule stated in
capitals a few lines above it.

The drop recovery added by the previous commit ran cn1GcRetainAllReferences after
gcSatbActive had already been cleared. Retaining MARKS referents that were white
and gcMarkDrain then scans them, so those objects are grey at a moment when no
barrier is watching -- and a mutator moving an old child out of one of them in
that window logs nothing on either side, leaving the child unmarked, not fresh,
and reachable only from a grey object the sweep will not protect. That is exactly
the hazard the trial-clear comment above describes, and the reason the clear is a
TRIAL rather than the end of the mark.

Recovery is one more thing that can turn out to mark something new, so it now
behaves like the catch it sits next to: re-arm gcSatbActive, retain, and go round
the fixpoint again rather than draining underneath a lowered barrier. At the
reopen cap it falls through to the same weaker invariant the cap already
documents.

recoveredDrops is what makes that terminate. Comparing against
cn1RefDropsAtCycleStart would re-trigger the recovery on every pass, because that
baseline never moves once a drop has happened -- the loop would then re-arm and
retain until it hit CN1_SATB_MAX_REOPENS every time a single drop occurred.
Recording the count each recovery consumed means another pass happens only if a
NEW drop has since been recorded.

Gates: run-gc-verify.sh green with all three self-tests firing, run-gauntlet.sh
green with all eleven tortures bit-identical and both GC stop modes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Track emergency slots when deciding the retain fixpoint

Accepted review finding. cn1GcRetainAllReferences decided it had converged by
comparing cn1RefDiscoveredTop alone, but discovery has two ways to make progress
and that counter only sees one of them.

During an emergency cycle the drain can reach another soft reference whose
referent cn1GcDiscoverReference clears on the spot -- and that path deliberately
does NOT append to the discovery list, because it exists precisely for the case
where the list could not grow. It fills a cn1RefEmergencyCleared slot instead. So
the length could be unchanged while a fresh recovery slot had just been written,
the loop would read that as "nothing new" and return, and the referent recorded
in that slot would never be marked. Reached from the CN1_SATB_MAX_REOPENS
fallback, that leaves the sweep free to take a referent a concurrent
Reference.get() is being handed.

The loop now watches cn1RefEmergencyTop as well and continues while either
counter moves.

Gates: run-gc-verify.sh green with all three self-tests firing, run-gauntlet.sh
green with all eleven tortures bit-identical and both GC stop modes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Consume the touch stamp when the emergency path retains an unrecorded reference

Accepted review finding, and it is a livelock inside the path that exists to
prevent one.

The emergency clear refuses to act on a reference whose stamp reads
CN1_REF_TOUCHED and marks the referent instead, which is right for this cycle: a
mutator may be holding it. But the ageing loop walks cn1RefDiscovered, and a
reference reaching that branch is there precisely BECAUSE it could not be
recorded in that list -- so nothing ever resets the stamp. It stays TOUCHED for
the life of the process, every later emergency cycle refuses to clear the same
referent however long ago it was last read, and if that retained memory is what
is blocking the allocation then codenameOneGcMalloc's retry loop never makes
progress. The emergency was raised by an allocation failure; this is the failure
mode it was added to break.

Consuming the stamp after marking closes it. The referent has just been retained,
which is the whole of what the stamp was protecting, so the next cycle is free to
clear it if nothing reads it again -- and if something does, that get() stamps it
afresh. Compare-exchange rather than a plain store so a get() landing between the
read and the reset is not silently discarded.

Gates: run-gc-verify.sh green with all three self-tests firing, run-gauntlet.sh
green with all eleven tortures bit-identical and both GC stop modes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Export SoftReference to the supported API, and retain under the barrier at the cap

Two accepted review findings. The first is a hole in this branch's own plan that
every gate here was structurally unable to see.

SOFTREFERENCE WAS IMPLEMENTED AND UNUSABLE. It existed only in vm/JavaAPI, but
maven/java-runtime builds the SUPPORTED API SURFACE from Ports/CLDC11/src and
BytecodeComplianceMojo indexes that artifact as the set of types an application
may touch. Application code would therefore resolve SoftReference against the
host JDK and then be rejected as forbidden API -- a confusing way to discover
that a shipped feature was never exported. The plan for this work said to add the
CLDC11 stub and it was never done; nothing caught it because every gate here
compiles vm/JavaAPI directly, which is exactly the path that bypasses the
compliance surface.

RETENTION TRACED WITH THE BARRIER DOWN AT THE REOPEN CAP. The drop-recovery path
was corrected for this a commit ago and the identical defect was left at
CN1_SATB_MAX_REOPENS: cn1GcRetainAllReferences marks referents that were white
and gcMarkDrain then traces them, so with gcSatbActive already lowered those
objects are grey and unwatched, and a mutator moving an old child out of one logs
nothing on either side. Retention now runs BEFORE the barrier is lowered, so
everything known at that point is traced under it.

What remains after the final take -- retaining references that last drain
discovered -- does trace with the barrier down, and that is said plainly at the
call rather than glossed: it is the same weaker invariant the cap already relies
on for the gcMarkDrain immediately above it, not a new exposure, on a path whose
own comment measures 0-4 reopens against a cap of 32.

Gates: run-gc-verify.sh green with all three self-tests firing, run-gauntlet.sh
green with all eleven tortures bit-identical and both GC stop modes; the CLDC11
java.lang.ref package compiles.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Resolve unrecorded referents before reading them, and stop self-test3 going stale

Two accepted review findings, both introduced by this branch in the last few
commits.

A NATIVE CRASH ON THE EMERGENCY PATH. Deciding whether an unrecorded referent was
already live read its mark word directly, without the cn1ConservativeResolve
guard the clear pass carries. Under conservative roots a dead Reference kept alive
by a stale native-stack word can hold a referent swept in an earlier cycle, whose
memory is now unmapped -- and gcMarkObject's own comment says reading even the
mark word of such a pointer faults. So a collection under memory pressure, which
is the only situation that reaches this path, could take the process down. The
referent is now resolved before any dereference, and an unresolvable one is
treated as neither live nor clearable: nothing can validate it, and it is either
garbage or an object allocated after this cycle's extent snapshot that the grace
rule keeps anyway.

SELF-TEST3 COULD PASS ON A STALE BINARY. It built RefPolicy-verify only when the
file was absent, while every driver above it rebuilds unconditionally -- so a
regression in the reference-verifier hook could still produce a green self-test by
running an old binary. That is the same "a gate that cannot fail" problem this
self-test was added to solve, reintroduced in the way the self-test itself is
built. It now rebuilds every invocation.

Gates: run-gc-verify.sh green with all three self-tests firing, run-gauntlet.sh
green with all eleven tortures bit-identical and both GC stop modes;
CN1_GC_CONFORM and CN1_DISABLE_BIBOP both build.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Delete the duplicated retain walk, and make the self-test rebuild fail loudly

Two accepted review findings, and both are the same mistake: a fix applied to one
copy of something and not to its duplicate.

THE EARLY DROP RECOVERY WAS A SECOND COPY of the retain-to-fixpoint walk. The
shared helper was taught that discovery advances TWO counters -- the emergency
path clears a referent into cn1RefEmergencyCleared without touching
cn1RefDiscoveredTop, because it exists precisely for when that list cannot grow
-- and this copy was left comparing the list length alone. It therefore read
"nothing new" over a freshly written recovery slot and returned without marking
what that slot held, so a racing get() whose enqueue had also failed could be
handed an object the sweep then freed. The copy is deleted and the helper called;
patching it would have left a third place to drift.

THE SELF-TEST REBUILD STILL ACCEPTED STALE CODE. Rebuilding unconditionally was
one of three things needed and the only one done. translate-and-build.sh replaces
its output only after the final compiler run succeeds, so a FAILED rebuild leaves
the previous binary in place; and `|| true` discarded the status, so the -x test
below accepted that binary. The self-test could then run old code and report
green -- the "gate that cannot fail" problem it exists to prevent, for the second
time in how it is built. It now removes the output first, checks the build
status, and fails the gate.

Gates: run-gc-verify.sh green with all three self-tests firing, run-gauntlet.sh
green with all eleven tortures bit-identical and both GC stop modes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Let drop recovery reclaim what the emergency condemned, and delete the last copy

Two accepted review findings.

DROP RECOVERY DEADLOCKED THE ALLOCATOR. Retaining everything on an SATB loss is
the safe reflex and it is wrong under the emergency budget: that budget is raised
by an allocation FAILURE, sustained exhaustion is exactly what keeps the SATB
stack from growing, and if soft-referenced data is what exhausted memory then
every retry cycle loses a batch, retains the same data, and codenameOneGcMalloc
spins on collections that free nothing. The emergency exists to break that
deadlock and this recovery was reinstating it.

The distinction that resolves it: the emergency decision never depended on the
log. "Drop every soft referent" is a policy choice taken from the memory budget at
cycle start, not an inference from liveness, so a lost log entry does not
invalidate it. What the log would have protected is a referent a mutator is
mid-read of -- and the touch stamp records that independently and
allocation-free, which is the signal sub-pass A already trusts. So
cn1GcRecoverAfterDrop retains touched referents, still clears condemned soft
ones, and retains everything else. The residual, stated at the code, is a get()
that loaded a soft referent and was descheduled before stamping: the window the
emergency path already accepts, against an allocator that otherwise cannot
progress.

A THIRD COPY OF THE FIXPOINT. The previous commit deleted one hand-written copy
of the retain walk and said a third would drift again; there already was one, in
the post-clear recovery, and it was not looked for. It had drifted the same way
-- watching cn1RefDiscoveredTop while the emergency path advances
cn1RefEmergencyTop -- so it read "nothing new" over a freshly written recovery
slot. There is now one implementation and no copies.

Gates: run-gc-verify.sh green with all three self-tests firing, run-gauntlet.sh
green with all eleven tortures bit-identical and both GC stop modes; the
emergency still reclaims under injected allocation failure (retained=0 at
softBudget=-1).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Quiesce before reading touch stamps in drop recovery, and spare live referents

Two accepted review findings, both on the recovery helper added one commit ago.

THE TOUCH STAMP WAS READ TOO EARLY. cn1SatbDrops becomes visible when an enqueue
FAILS, and that happens before the accessor reaches its stamp -- so entering
recovery on that signal and reading touchAgeField immediately can see "not
touched" for a getter that is mid-load and about to stamp. Clearing on that
reading hands the sweep an object the getter is being given, and this path
deliberately does not record clearedReferent, so nothing downstream could recover
it.

The helper now quiesces first. Every getter registers across its whole load, the
stamp included, so an in-flight count of zero means every getter that overlapped
has finished and published -- which is what makes the stamp evidence rather than
a race.

A STRONGLY REACHABLE REFERENT COULD BE CLEARED. The other emergency path consults
the mark word before clearing, because a SoftReference may only be cleared when
its referent is SOFTLY reachable; this helper omitted that test and would clear an
application's cache entry for an object it also holds in an ordinary field. It now
applies the same current-epoch-or-fresh check.

Both are the same shape: new code on one path missing a guard its sibling already
carries.

Gates: run-gc-verify.sh green with all three self-tests firing, run-gauntlet.sh
green with all eleven tortures bit-identical and both GC stop modes; the emergency
still reclaims under injected allocation failure (retained=0 at softBudget=-1).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Guard the recovery's header read, and record what it clears

Two accepted review findings, both on the helper this branch added three commits
ago, and the first is the previous fix creating the next defect.

THE MARK-WORD READ HAD NO RESOLVE GUARD. It was added one commit back to stop the
recovery clearing strongly reachable referents -- and it dereferences the referent
without cn1ConservativeResolve, which the clear pass and the unrecorded emergency
path both apply before the same access. A Reference kept alive by a stale
native-stack word can hold a referent swept in an earlier cycle whose memory is
unmapped, and reading even its mark word faults. So a fix for a correctness gap
introduced a native crash, in code whose two siblings show the right pattern.

THE QUIESCE CANNOT HOLD. It drains the getters in flight when recovery starts and
cannot stop a new one registering immediately afterwards, while gcSatbActive and
gcSatbTerminating are both still raised. That getter can load the referent, have
its enqueue fail, and be descheduled before stamping, so this loop reads the old
stamp and clears a field whose referent is being handed out -- with nothing saved,
no later pass could mark it.

The clear is now recorded. The post-clear recovery therefore marks it whenever a
drop is visible, which gives the emergency's reclaim back in exactly the case
where safety is uncertain, and keeps it in the common case where no further drop
occurs. That is the right way round: reclaim is the goal, not being right.

This is the fourth consecutive round on this one helper, each fix producing the
next finding. The oscillation is between two poles this path genuinely sits
between -- reclaiming under memory pressure, and staying safe against an
unreliable log -- which is an argument about the shape of the design rather than
about any of the individual fixes.

Gates: run-gc-verify.sh green with all three self-tests firing, run-gauntlet.sh
green with all eleven tortures bit-identical and both GC stop modes. The first
gauntlet run was killed at IbpTest and re-run from the start; a partial gauntlet
is not a result.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Do no reference work in a program that has no references

Localises and fixes an intermittent CI regression, and removes a cost every
application was paying for a feature most of them never use.

BibopPageFloorIntegrationTest began failing on the 4-marker arm64 job -- five
times against master's five clean samples -- reporting its pages as released
while phys_footprint stayed at ~231MB. The workload contains no Reference at all,
which is why it took so long to believe: nothing in this feature can run there.

Except one thing did. cn1GcProcessReferences ran on every outer termination pass
regardless, and once the drop re-check landed it called cn1SatbBulkQuiesce()
unconditionally -- which is not free. It spins in usleep(50) while any BULK ARRAY
COPY is in flight, and that app copies object arrays, so the collector could stall
inside the termination loop on behalf of a feature the program does not use. It
fits the shape that never made sense otherwise: intermittent, only under the
configuration where four markers and bulk copies actually overlap, and reporting
release while the footprint does not move.

Dispatching the workflow on wip/refbisect-base -- master plus only the first
commit, weak references with none of this machinery -- passes the same job, which
is what localised it to the later commits rather than to the feature.

The fix is what should have been there from the start: with nothing discovered and
nothing in the emergency array, the pass returns immediately and touches nothing.

An earlier hypothesis for this failure was WRONG and is recorded so it is not
retried: cn1RefBeginCycle's headroom probe was suspected of adding a per-cycle
footprint syscall, but the test sets only CN1_LOG_PAGE_RELEASE, never
CN1_SIMULATE_PROC_MEMORY_LIMIT, and cn1SimulatedProcLimitBytes caches -- so on
Linux that path returns -1 from a cached atomic read.

Gates: run-gc-verify.sh green with all three self-tests firing, run-gauntlet.sh
green with all eleven tortures bit-identical and both GC stop modes; RefPolicy's
weak, alias and cache assertions unchanged.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Nursery promotion, two silent gates, and a baseline the rename broke

cn1GcDiscoverReference now defers to gcMarkObject while a thread is inside
its own nursery minor collection. cn1PromoteDrain runs the generated mark
functions with nurseryPromoting raised, and gcMarkObject's promotion branch
is what moves a referenced object out of the block being recycled -- routing
the referent past it meant a surviving WeakReference could be promoted alone
and left pointing into a reclaimed block. Nothing clears on that path, so the
edge costs a promotion and nothing else.

The nursery arm did not compile at all, here or on master: nativeMethods'
bulk barrier calls cn1SatbBulkBegin unconditionally while the prototype sat
in the #else of the CN1_NURSERY split, which clang rejects as an implicit
declaration under C99. The load barrier added a second instance. Declared
beside the deletion barrier instead, where the callers are.

Two gates that could not fail:

- RefPolicy printed WEAK_LIVE_KEPT and exited 0 whatever it said. A referent
  cleared while still strongly reachable is heap-SAFE -- null dangles nothing
  -- so run-gc-verify cannot see it either, and the checksum reads live[]
  directly. Both advertised validation paths could stay green while
  WeakReference silently emptied every cache built on it. It is an assertion
  now.
- ab-refs.sh ignored the return code. Metrics print before exit, so a VM that
  corrupted its heap and died in an atexit handler still emitted RESULT and
  the whole table, and the harness published checksum-matched medians from a
  crashed run.

The cast-semantics baseline names anonymous classes as Outer$N, so #5746
adding one to AndroidImplementation renumbered onReceive from $46 to $47 and
left the entry stale. master is red on the gate today; this is the same cast,
not a new one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* The emergency reclaim retained everything it condemned

cn1GcRecoverAfterDrop is reached precisely when soft-referenced data is what
is exhausting memory, and blanket retention there is what lets the allocator
spin -- which is why it clears selectively instead of calling
cn1GcRetainAllReferences. It then recorded each condemned referent in
clearedReferent and marked that same referent one statement later in the same
iteration, so the sweep kept every one of them and the reclaim never happened.
Its own comment described the behaviour it did not have: "in the common case,
where no further drop occurs, the clear stands and the reclaim happens".

The recovery is owed to clears made by an EARLIER pass, which predate the drop
that brought this one here; it is not owed to the clears this pass has just
decided for itself, after its own quiesce. Entries therefore record which pass
cleared them.

A drop count cannot make that distinction, and the first version of this fix
used one. The getter the record defends against fails its enqueue BEFORE the
clear stamps anything, so the count at the clear already includes it, and
"the count moved since" is false in exactly the case where the mark is owed --
it would have reintroduced the dangling read the record exists to prevent.
The drop count is now read once per pass, after the quiesce that bounds the
window, and a drop past it retains through the shared walk.

Two harness gates that could not fail:

- RefPolicy exits nonzero on ALIAS_SPLIT. This does NOT make the phase a
  detector for the violation, and the code says so: with
  -DCN1_REF_NO_ALIAS_ATOMICITY putting the single-loop bug back, three runs
  still reported 0/256, because catching it needs a get() inside a window
  microseconds wide. The ablation is what tests that path.
- ab-refs.sh requires the weak phase to have cleared something -- and for
  noweak, nothing. The checksums are independent of retention policy by
  design, so they agree just as well when an arm keeps every referent strong,
  which is the regression that matters.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Record why the porting layer still hands back a WeakReference

A review read the unmigrated createSoftWeakRef as the feature being undelivered:
the base implementation still returns a WeakReference and the iOS override still
pins every entry in a Hashtable until a memory warning replaces the map, so no
framework cache constructs the new class.

The facts are right and the conclusion is not. Migrating them changes the
lifetime of every decoded image, gradient and resource cache in every app on
every platform, which wants its own change and its own bisect point -- and each
call site needs deciding rather than sweeping, because a lifetime tracker like
JavascriptContext reads a null extract as proof of collection and breaks under a
reference that outlives its referent. This change is the mechanism and the
measurement that justifies it.

Noted at both places a reader arrives from, since a PR thread is not somewhere
anyone looks later.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant